In order to analyze the complexity of space borne optical remote sensing system and the source of complexity and its forms of optical remote sensing system were determined by analyzing the whole imaging chain, and the mathematical model of this process was built starting with the basic characteristics and rules of complex system. Based on the important parameter, namely MTF (modulation transfer function), the fundamental research pathway of complex system was analyzed by combining mathematical modeling and computing, namely combing principle reductionism with principle holism and combing qualitative analysis with quantitative analysis. Through the contrasting of the mathematical model and the computing result, on the one hand, the relationship between system performance and these key parameters were figured out; on the other hand, the true MTF of the system was obtained by using regularization blind deconvolution method. These analyses can make great contribution to not only the image quality but also the optimization of remote sensing system.