To mitigate the degradation of tracking accuracy induced by numerous false association ghost points, this paper introduces a dual-level ghost point elimination and target tracking algorithm, which integrates angular measurement with target motion characteristics. The proposed algorithm employs a cooperative localization strategy that prioritizes association before estimation. By establishing a mapping relationship between angular measurement noise and localization error, a field-of-view grid map and an energy accumulation matrix are constructed. Through a detailed analysis of the spatial geometric distribution characteristics of real targets and false association ghost points within the field of view, a novel elimination criterion based on Hough transform theory is developed, facilitating the primary coarse elimination of ghost points. Additionally, by examining the distribution characteristics of target localization ambiguity regions and motion features, a predictive tracking gate is constructed using motion parameter identification, enabling the secondary fine elimination of ghost points at the kinematic level. Experimental results demonstrate that the proposed algorithm significantly enhances target tracking accuracy, achieving a ghost point elimination rate of 91.82%, thereby effectively addressing the false association problem in multi-target tracking.