Gabor transform and K-means algorithm are two commonly used texture analysis methods. However, the texture feature vector has a high dimension by using Gabor transform, which will influence the operating efficiency. Meanwhile, K-means algorithm is affected by the initial clustering centers, and it may lead to the decrease of classification accuracy. Although, some optimization algorithms like genetic algorithm and particle swarm optimization algorithm could improve the performance of K-means algorithm to some extent, the optimization effect is difficult to guarantee as the increase of dimension. Hence, the Relief algorithm was applied to make a feature selection for Gabor texture feature, and to obtain a suitable texture feature subset. Furthermore, a differential evolution algorithm was used to optimize the clustering center of K-means algorithm, and enhance the accuracy and efficiency of texture recognition. Experimental results demonstrate that the dimension of texture feature vector by using the proposed method is obviously lower than that by using the original feature set, and the recognition accuracy is also apparently improved than the basic K-means algorithm.