Tomato quality is one of the most important factors ensured the consistency of tomato market factors. A color analysis method was proposed for classifying the fresh tomato, with reference to the national standard GB 8852—88, defining the classification standards of tomato maturity. In this study, tomato was divided into the four categories, full ripe, ripe, half ripe, and green ripe. RGB images of tomato were collected, removing the background and filtering de-noising, and then they were converted to HIS and HSV color models. Through the MATLAB programming, the mean values of the color components R, G, B, H, S, V, and I were obtained, and the determination and selection of combination components were carried on using SPSS software. Moreover, the discriminant analyses were then performed using Matlab. The results showed that the identification rates of the green ripe and validation sets were the best of all the three different discriminant functions, reached 100.00%, and the highest discrimination rates of half ripe tomato set was 94.74%. However, the identification rates of training and validation sets of ripe tomato were the lowest, identified as 76.67% and 70.00%, respectively. Those of the training and validation sets of full ripe were the highest, about 90.00%. In general, the discrimination and classification of tomato with different maturity were realized using the machine recognition system in the present study.