Single Channel Blind Signal Separation (SCBSS) is a hot and difficult topic in signal processing field. The feasibility of cyclostationary filtering applied in the single channel signal separation problem for the spectrally overlapping signals was analyzed, and then a separation algorithm based on frequency shift filter and Schmidt orthogonal cancellation was proposed. Firstly a source signal by frequency shift filter was extracted, then another source signal was estimated by subtracting the signal recovered from the mixture based on Schmidt orthogonalization. Simulation results indicate that, compared with the Wiener filtering technique, the proposed algorithm can separate radar and communication signals which are spectrally overlapping effectively and the proposed algorithm estimates the source signals with higher gain about 10dB. 