We proposed a method of defect detection for jujube based on a neural network with network-in-network convolutional (NIN-CNN). This method adds 1×1 hidden perception layer to the original AlexNet convolution neural network; enhances the non-linearity of the network to extract more abstract features; and replaces the fully connected layer with the global average pooling layer, which improves the recognition accuracy while reducing a large number of parameters. In this study, Jun jujube in Xinjiang is tested. The jujube is divided into seven categories, including healthy jujube, black-spotted, wrinkled, overlapping, peeling, yellow-skinned and crack. The experimental results show that the classification effect of the proposed method is improved effectively, compared with the conventional visual detection method with SVM and the classification method with AlexNet network.