In using the combination system with visual and IMU (inertial measurement unit) to estimate the relative pose of non-cooperative target in space, the external parameters of the system have a great influence on the estimation accuracy. Considering the complex and time-consuming calibration of the external parameters, a method to estimate the relative pose and calibrate the external parameters simultaneously by using visual and IMU combination system was presented. This method takes the external parameters as the state variables and forms the system state equation together with the relative orbital kinematics equation, the relative attitude equation and the IMU model. Then, the relative pose, IMU biases and external parameters of the visual and IMU were estimated by using the extended Kalman filter designed with the state equation and the observation of the monocular vision. The validity of the method was verified by mathematical simulation. The simulation results show that this method can estimate the relative pose and IMU biases effectively and calibrate the external parameters, when the deviation of external parameters existed.