In this paper, an algorithm based on deep residual network is proposed to recognize and classify the surface defects and texture of jujube. The algorithm adopted jujube’s G channel of RGB color figure then to get the characteristics of the figure as network input, using residual learning way to expand the depth of the neural network learning, and residual error of the neural network activation function Relu replaced with SELU, the loss function softmax loss with center loss to replace, Dropout layer was developed for the training, reduce the risk of network through fitting, solved with deepening study depth gradient dispersion and explosion phenomenon in the network. The results showed that the classification accuracy reached 96.11% and the detection efficiency was about 120 min 1.