In order to enhance the anti-jamming operational capability of cognitive antijamming system based on simultaneous transmitting and receiving in the complex electromagnetic environment, an improved energy detection scheme based on the system was studied and the closed expressions of detection probability, false probability and outage probability of the system under Rayleigh fading channels were derived. The scheme decides the detection threshold by obtaining the result of comparison between the selfinterference channel capacity and the required transmission speed for receiver. Simulation results show that the effect of receiver impacted by selfinterference can be reduced and the outage performance of the system can also be improved by the proposed detection scheme.