Infrared cameras are suitable for complex environments, and the use of infrared images to detect black-flying UAV targets has important application value. Aiming at the problems such as small size of UAV target, few pixels in the image, weak texture detail information, and the difficulty of the algorithm to effectively extract the infrared UAV target features resulting in low detection accuracy, this paper proposes a target detection algorithm with multi-scale learning. By constructing a multi-scale feature fusion structure in the neck network of the model, introducing a multi-scale feature learning module, cascading the features of the deep network and the shallow network, acquiring the features of the target at multiple scales, enriching the semantic and feature information of the feature map, the algorithm significantly improves the accuracy of the detection of the target of small UAVs. The SIoU is used instead of the CIoU loss function in the training process, which minimizes the loss of the network model in the training process and improves the regression accuracy. The experimental results show that compared with other infrared small targets and mainstream detection algorithms, the method proposed in this paper can effectively improve the detection accuracy of UAV targets, and can meet the detection accuracy requirements for detecting UAV targets in practical applications.