A color and gradient feature based image segmentation and recognition algorithm is proposed for Mycobacterium Tuberculosis (TB) objects in microscopic images. First, the input color image was pre-segmented based on the HSV color space using the threshold method according to the prior information. Then, the original image was transformed to the CIE L*a*b* color space, and the L component image was segmented using an adaptive threshold method to get finer segmentation result. In order to accommodate the complex variety of image background, all the suspected objects were verified according to the “hue consistence” assumption and the false objects were rejected. To identify the TB bacilli, the algorithm used five shape feature descriptors, including the area, the perimeter, the ratio of width to height, the compactness and the roughness, and two gradient feature descriptors, the gradient magnitude weighted average (GMWA) and the gradient magnitude variance average (GMVA) respectively, and makes the judgment through Bayes classifier. Experimental results show that the proposed algorithm can accommodate the complex variety of specimens and the image background, and a high recognition rate (91%) can been obtained.