Aiming at the moving objects in the video sequences collected by the mobile single camera, an online multi-object automatic tracker was focused on research, which was based on the relative motion information and data association strategies. The recovery of the object trajectory was achieved by the relative motion model between the objects, and the object trajectory fragmentation was reduced. The assignment between detection of the current frame and the past trajectories was improved based on the event matching algorithm, which reduced the number of identity conversions during tracking. Experimental results show that the improved algorithm is more accurate than the original algorithm in tracking and positioning the object in the sequences, reducing the number of trajectory fragmentation and identity conversion, and our tracker achieve relative the same performance in the state-of-the-art on the TUD-Campus sequence in the international academic circle.