Objective: To solve the current situation of manual screening in most cases of Chinese apple classification. Methods: The infrared and visible image fusion algorithm based on multi-scale transformation was used to fuse the collected visible image and infrared image of the apple to obtain a more intuitive fusion image with defect characteristics, and performed image preprocessing operations on the image to obtain a binary value. The image data set was transformed, and then the AlexNet model of the convolutional neural network was used to train, verify and detect the previous Apple surface defect data set. Results: The detection method has an average accuracy of 99.0% for intact fruit, defective fruit, calyx/stalk, calyx/stalk plus defect on the produced apple surface defect data set, and the average accuracy was 99.0%. The recognition accuracy rate could reach 95.8%, and the recognition accuracy rate of intact fruit, defective fruit and calyx/fruit stem plus defect was as high as 100%. Conclusion: This method has a relatively high detection accuracy for apple surface defects, which can meet the demand for online classification of apples.