Objective: This study aimed to optimize and improve the CenterNet method for detecting fruit defects. Methods: the lightweight convolutional neural network of MobileNetV3 was used to replace the original backbone network of CenterNet accelerate the detection speed, improve the module of MobileNetV3, enhance the detection ability of the model for small and medium-sized defective blocks of fruit, and optimize the pre detection stage of CenterNet to increase its detection accuracy. Results: The recognition rate of significant defects such as apples with diameter > 4 mm was 99.7%, and the detection speed was 113 FPS, with the model volume of 1.31 MB. Conclusion: Compared with models CenterNet _ Resnet18 and CenterNet_Shuffler, model MO-CenterNet has better balance in training time, detection speed and accuracy.