The main errors of tri-axial magnetometer include scale deviation error, sensitivity error and tri-axial non-orthogonal error. The non-orthogonal error of the sensor was analyzed under tri-axial orthogonal coordinate system. The sensor error model was established, and the influence of error on the shaking of the sensor was analyzed. Shake 1° to produce the error up to 109.5 nT, the error increases with the increase of the shaking angle. To control the shaking error, the calibration model was established, and through the nonlinear curve fitting to estimate the model parameters of the calibration model accurately. After compensation, the sensor output error was greatly reduced, and the shaking error can be controlled below 0.03 nT, which shows that this method can effectively reduce the influence of shaking on sensor output.