A method of automatic building height extraction by Fusing Catadioptric Panoramas and Remote Sensing Images was presented for solving the problem about building height extraction in 3D city reconstruction. The method has three steps. First, images between Catadioptric Panoramas and Remote Sensing were registered. Second, the building in the Catadioptric Panoramas were extracted by using the previous result of image registration, Finally, the building height with catadioptric projection model was calculated based on the information of the top and bottom borderlines. The whole processing of building height extraction was completed automatically. Experiments show that this method is feasible and can obtain a precise result, especially for dwelling houses, error reaches no more than 4%.