In this study, three kinds of disease images of jujubes, black spots, broken heads and dry strips with high classification difficulty were used as research materials. The color moment and gray level co-occurrence matrix were used to extract 14-dimensional eigenvectors of the color and texture features of jujube, and the principal component analysis method was used to optimize the features. Four principal factors of eigenvectors were obtained and then used as the input of support vector machine. The crossover algorithm was used to determine the optimal support vector machine penalty parameter c and kernel function parameter g, which was used as the parameter of the support vector machine multi-classification model to train the model. Using the trained model to perform multi-classification experiments on the jujube, the results proved that the three kinds of defects of jujube could recognized quickly and accurately, with the recognition rate at 93.3%, 100.0% and 96.6%, respectively. The classification accuracy of this model for jujube defects could reach 97.2%, with high efficiency.