Since the workers have significant differences in the knowledge level and evaluation criteria, the quality of the collected labels varies a lot. It′s of key importance to improve the quality of labels and learning models in crowdsourced label learning. A novel double-confidence inference algorithm was proposed to solve the problem of crowdsourced label inference. The workers′ confidence was obtained via the data distribution characteristics and label information, and then the label was inferred by this confidence so as to improve the quality of the integrated label. The experimental results show that the proposed algorithm outperforms other ground truth inference algorithms only based on label information.