A new nonlinear filter algorithm for micro satellite attitude determination was proposed, which adopted three-axis magnetometer(TAM) and fiber optic gyroscope(FOG) as attitude sensors. In the design of the nonlinear filter, square-root sigma point Kalman filter was modified. Firstly, an augmented state vector was derived by combining the vector part of attitude quaternion, the bias and noises of FOG. Secondly, vector rotational model, optimization model and error quaternion multiplication model were established to guarantee the quaternion normalization constraint in the process of nonlinear filter. The simulation results indicate that attitude determination performance is improved effectively by the presented algorithm. In comparison with EKF, the accuracy and stability of the proposed algorithm is much better, and the convergent speed is faster. In comparison with UKF, the convergence is equivalent, the accuracy is slight better, while the stability and computing efficiency are both higher.