In view of the problem that it is difficult to predict the inner pulp defect of banana, the machine vision technology is employed to recognize the image of banana peel and pulp, and then fits the recognition parameters to get the prediction model of pulp defect. The collected image, grayed and filtered, is recognized by double threshold and morphological analysis to extract banana peel, banana pulp, banana peel black spot and banana pulp defect. Thereafter, the total number of pixels in the extracted region is calculated, and the total number of pixels is taken as the area of the region. The ratio of the total area of banana peel / total area of banana pulp to the area of black spot of banana peel / defect area of banana pulp is used to define the degree of black spot of banana peel and defect of banana pulp. Using polynomial fitting method, the prediction function of pulp defect is obtained according to the training samples, and the residual analysis is carried out. The accuracy of banana grading was 88.9%. Compared with the other method to predict the defection of class by peel of banana, the accuracy of prediction mathematic is better, indicating the more practice value of predicting by flesh of banana in this study.