Aiming at the high difficulty and high false alarm rate of infrared dim small moving target detection in low altitude and complex scenes, a precise detection method based on full convolution network and a low-latency parallel processing method based on FPGA (field programmable gate array) were proposed to meet the real-time processing application requirements of high frame rate images in the detection system. A lightweight full convolutional network was used to detect dim small targets in spatial domain, and time domain trajectory correlation on suspected targets in adjacent frames was performed to further reduce false alarms. The experimental results show that the detection rate and false alarm rate are significantly improved compared with the five traditional methods. And the real-time processing of 100 Hz image is completed on a single FPGA with the processing delay less than 1.8 ms. This method realizes high-precision, high-robust and fast real-time detection of dim small targets in complex low altitude scene.