Particle transport simulation is one of the main applications of high performance computers. But facing to its fast growing compute requirements, the general-purpose microprocessors cannot adapt to the particle transport program features, owing to the complexity architecture of its single core, and then it is difficult to obtain high ratio of performance and power. Therefore, the program features of the particle transport non-deterministic numerical simulation were extracted and analyzed. Based on the characteristics of the algorithm, the architecture of accelerator core was designed, including pipeline structure, branch prediction unit, multi-level Cache hierarchy and main memory design. A specific accelerator architecture was designed in accordance to the particle transport program features. The simulation results of running the particle transport program on the general architecture simulator show that, as compared with ARM Cortex-A15, the proposed specific accelerator can achieve 4.6 times performance improvement under the same power consumption, and 3.2 times under the same area.