In order to study the self-excited vibration of rocket in the flight test, a two-degree-of-freedom linear system based on the coupling of elastic force and aerodynamic force was established, and the displacement feedback between pitch and yaw degree-of-freedom maked the system obtain energy from the outside and cause the self-excited vibration. By analyzing the stability of the motion equation of the system, the criterion of losing asymptotic stability of equilibrium point and generating self-excited vibration was obtained. The time history curves of pitch and yaw angular displacement were obtained by solving ordinary differential equations with Runge-Kutta method. The comparison between the simulation results and the measured data shows that the vibration divergence phenomenon during the flight of rocket is the mode-coupling self-excited vibration caused by displacement feedback. In addition, the analysis results show that the beat or constant frequency vibration usually occurs before and after the self-excited vibration.