Backward wave oscillator with over-mode Ka-band high power microwave (HPM) generally outputs mixed TM0n modes, it has high energy conversion efficiency, but it can′t ensure the purity of output modes. However, it is necessary for high radiation efficiency that the microwave input to the radiation terminal is certain single and pure. So it is hard to put these HPM sources into application. In order to solve this problem, a design method for compact hybrid modes converter of purifying TM0n mixed modes was proposed. With this method, the TM01, TM02 and TM03 mixed modes can be converted into pure circular waveguide TM01 mode efficiently on the condition of high power capacity and wide wave band. This method also reduces the modes purification difficulty in the design of the HPM sources.