The nonlinear effect of phase performance and the dependency relationship of envelope delay with the change of frequency for a radio frequency module are owed to a good many factors uncertainty, which may cause the waveform of output radio frequency signal to be distorted. Based on the recursive least square adaptive filter, the novel method to eliminate nonlinear effect of radio frequency module was proposed. By regarding the GPS simulating system as an example, the model parameters were estimated precisely by using its self-revising module, software radio technique and digital signal processing technique. Then in order to eliminate the nonlinear phase and variable envelope delay, the synthetic signal in baseband was processed using a digital filter, which was the inverse filter of radio frequency module. The corresponding computation was accomplished utilizing the self-programming emulating package with the help of Matlab. The simulation results indicate that the method proposed in this paper can eliminate the impact on radio frequency signals by the nonlinear effect of phase, and the waveform distortion of output signals can be reduced to less than 0.01 percent.