A calibration approach based on the invariant of interstar angles was put forward for star trackers. Compared with common approaches, the sine of interstar angle which provides greater sensitivity for the angle changes closed to 0° was adopted to construct the observation equation instead of its cosine. In the calibration procedure, the two-step approach was proposed to estimate the principal point, focal length, and distortion parameters. The iterative optimization was also proposed to get the optimal estimations. In order to improve the accuracy of calibration, the apparent position of star at the observing time was modified. The experimental results indicate that the sine method is more accurate; and the deviation root-mean-square of the measured interstar angle is 2.2×10^-5 rad after calibrated.