Multi-sensor image matching is a challenging problem in image process field. As synthetic aperture radar images and optical images have significant differences, most existing methods cannot achieve satisfied matching result. To respond to this issue, a new multi-sensor image matching method based on Gabor binary encoding was presented: the big and small input images were first convoluted respectively by a group of Gabor filters; the compressed representation was executed on the convolution result by using pooling method; the binarization of pooling results was conducted and it was transformed into binary code to create Gabor binary encoding features; the similarities of corresponding window features between real-time images and reference images were calculated by using bit manipulation and the maximum value indicated the matching result. This method describes images by binary representation, so the computation complexity is much lower than that of the traditional method, while the common characters are better revealed. Experimental results show that the proposed method has much higher matching rate and require much lower computation time than those of the existing methods.