As for nonlinear/non-Gaussian information processing problems in navigation systems, a kind of adaptive integrated navigation system was established on the basis of the modified traditional nonlinear Kalman filter by utilizing self-organization algorithm, neural network and genetic algorithm. Applying self-organization algorithm with redundant trends, Volterra neural network and genetic algorithm, the nonlinear prediction model of navigation system error was built. Then, predicted values of navigation errors were obtained using the established error model. Comparing the predicted values with the estimated values by Kalman filtering algorithm, the difference between them, functioning as an indicator of the divergence of Kalman filter, was formulated. The modification of nonlinear Kalman filter was made and a novel technology of navigation error compensation was thus developed on the basis of adaptive control methods. Applying traditional and modified Kalman filtering algorithms respectively, the semi-physical simulation study based on the navigation system KIND-34 was carried out. The analyzed results indicate that the accuracy of error estimation and compensation in navigation systems is improved by using the modified nonlinear Kalman filter, and thus the ability of self-adaption and fault tolerance are enhanced in integrated navigation systems.