For the sake of tri-axial magnetometer calibration, an error compensation model of tri-axial magnetometer was established. The sine curve fitting method was proposed to calibrate the tri-axial magnetometer based on spin data. The calibration of triaxial magnetometer could be implemented by calculating the calibration parameters using the initial phase angles of the sine curves. The results indicate that it has high precision to calculate the calibration parameters of tri-axial magnetometer, and the measurement accuracy of calibrated magnetometer is improved significantly. It dose not require high-precision optical pumping magnetometer to measure the magnetic field intensity, thus it puts forward an effective way for calibrating tri-axial magnetometer.