Motivated by the demands of real-time video stabilization, a real-time video stabilization algorithm based on homography trajectory is proposed. For each input frame, our approach regenerates four short homography trajectories by applying inter-frame homography transformations to the four corners of image rectangle. An associate Kalman filter is then performed to smooth these transformational trajectories. Finally, at the stage of image composition, constraints of inclusion and similarity are considered for selecting a visually plausible stabilized video. The proposed method can offer real-time video stabilization and it can remove the delays for caching coming images. In addition, our method does not rely on sophisticated 3D reconstruction or long-range feature tracking and it can effectively relieve the errors introduced by using homography to model the video motion. Experiments show that our approach can offer real-time stabilizing for videos with 2D scenes or the 3D scenes with moderate depth variation. 