In order to intensive study the self-excited vibration phenomenon of rocket in the flight test, a two-degree-of-freedom linear system based on the coupling of elastic force and aerodynamic force was established, and the working mechanism of self-excited vibration caused by the displacement feedback between pitch and yaw degrees of freedom was explained. By analyzing the stability of the motion equation of the system, the criterion of losing asymptotic stability of equilibrium point and generating self-excited vibration was obtained. The time history curves of pitch and yaw angular displacement were obtained by solving ordinary differential equations with numerical method. The comparison between the simulation results and the measured data shows that the sudden amplification of vibration divergence phenomenon during the flight of rocket is the mode-coupling self-excited vibration caused by displacement feedback. In addition, the analysis results show that the beat or constant frequency vibration usually occurs before and after the self-excited vibration.