Recently spatial pyramid matching (SPM) with scale invariant feature transform (SIFT) descriptor has been successfully used in image classification. Unfortunately, the codebook generation and feature quantization procedures using SIFT feature have the high complexity both in time and space. To address this problem, a feature descriptor called Binary Pattern of Oriented Gradients is presented. Firstly, the input image was densely sampled and divided into small uniform image patches. Secondly, each patch was divided into 2*2 grids uniformly. For all grids the histograms of oriented gradient were computed and all dominant directions of the histograms were coded by binary coding. Then the descriptor was generated by converting the binary number to decimal number. Finally, this descriptor was combined in the spatial pyramid domain. Experiments on popular benchmark dataset demonstrate that the proposed method always significantly outperforms the popular SPM based SIFT descriptor method both in time and classification accuracy.