To solve the problem of non-linear gray level distortion and strong noise interference in remote sensing multi-modal image matching, a remote sensing multi-modal image matching algorithm based on PCA (principal components analysis) and HOG (histogram of oriented gradients) was proposed. This algorithm uses HOG to extract the common features of geometric structure between images, which can effectively overcome the problem of nonlinear grayscale distortion of multi-modal images. Besides, a fast multi-scale PCA algorithm was proposed to enhance the local gradient direction in HOG, so that it can accurately extract the structural features of the image under the condition of strong noise interference. In order to improve the calculation speed of the algorithm, the integrated image method was used to reduce the computational complexity of the feature extraction process, and the fast Fourier transform was used to achieve a highly efficient matching search. The experiment used a variety of remote sensing multi-modal images (including visible light images, synthetic aperture radar images, and infrared images) to verify the matching algorithm. The results show that, compared with existing algorithms, the proposed algorithm significantly improves the matching performance.