3D structure from aerial image sequence is acquiring much attention in aerial photogrammetry. A novel method based on factorization and bundle adjustment，as well as its principle and implementation is presented. Being the bridge between factorization and bundle adjustment, the notion “R-T”mode based collinearity equation is clarified as well as its linearization equations. Synthetic and real image test show the new method gives an accurate and robust result, and it can be applied to aerial surveying and mapping, virtual reality and military reconnaissance, etc.