An object tracking algorithm by associating tracklets with the best bidirectional spatio-temporal continuity was proposed. First, reliable tracklets were generated by a primitive frame-by-frame association; then tracklet's motion, position and color features were computed and refined by applying Kalman filter and weighted mean method respectively; finally, the best spatio-temporal association of tracklets was achieved through an iterative association by computing spatio-temporal continuity between tracklets. Experimental results prove that multiple objects can be successfully tracked under occlusion both by other object and scene object.