PCANet (principal component analysis network) is a simple deep learning algorithm with excellent performance in the field of image recognition. Integrating the idea of graph embedding into PCANet, a new image recognition algorithm Smooth-PCANet was proposed. In order to verify the effectiveness of the Smooth-PCANet algorithm, adequate experiments were performed on different data sets such as face, handwritten characters, and images. Compared with several image recognition algorithms based on deep learning, the experiments demonstrated that the Smooth-PCANet achieves higher recognition performance than the PCANet and avoids overfitting more effectively, with a significant advantage in small samples training.