In the content-based retrieval, the conventional approach to the retrieval algorithms is to conduct the feature extraction and similarity match on the basis of single feature and then to conduct multi-feature extraction under the condition of weighting the result of single feature extraction. Such an approach incurs large amount of calculation and lacks flexibility. Therefore, a new architecture to organize different retrieval algorithms in content-based retrieval is proposed. It appears as a hierarchical architecture, which is able to acquire a gradually accurate result. The transfer factor, priority factor and complexity factor are defined in this architecture, which can help to combine different feature-oriented retrieval algorithms flexibly. Experiment shows that this architecture improves the retrieval efficiency, resulting in a reduction of 40% of the retrieval time.