For surveillance trajectory, a potential semantic content discovering method based on motion features was investigated. In the phase of feature estimation, precise and smooth multi-modal distribution of direction was obtained by combining inflection points of trajectory curve with the kernel density estimation. Then sequential and concurrent temporal patterns of trajectory sub-class were modeled by Hidden Markov Model. With motion feature distribution, a hierarchical trajectory aggregation model based on motion similarity was proposed. Experimental results show that the model can be used to discover potential structure of trajectory set which reflect the motion region information of surveillance scene.