Objective: In order to classify the pecan materials after the shell is broken, and to improve the deep processing level of pecans. Methods: 5 types of pecan samples were obtained through the image acquisition system, including relatively intact shell kernels, undivided kernels, unbroken intact pecans, incomplete shell kernels undivided, and shells. Using the data augmentation way, a sample containing 15 000 images created data sets were obtained. Build a model based on the VGG16 network, which was trained and verified on a data set containing 5 types of pecan material images according to the ratio of 9∶1. Results: The results showed that the accuracy of model training and validation accuracy reached 97.3% and 99.7%, respectively. Through classification and recognition of 1 713 hickory processed material images, the accuracy reached 99.5%. Conclusion: The model can be achieved after a break of pecan shell material classification accuracy requirements.