Objective: In order to improve the situation that existing fruit recognition and classification methods rely on manual operation and complex equipment. Methods: A lightweight model YOLO-FFD (YOLO with fruit and freshen detection) was proposed, which based on the YOLOv5 framework. Firstly, LightweightC3 was designed as the basic unit of the backbone feature extraction network based on the depth separable convolution and GELU activation function, which reduced the number of model parameters and computation, and speeds up the convergence of the model. Secondly, EnhancedC3, a large kernel depth separable convolution module, was used to improve the neck of the original model, suppressed information loss and enhance the feature fusion ability of the model, so as to improve the detection accuracy of the model. Finally, GSConv was used to replace the common convolution in the feature fusion network to further lighten the model. Results: The experimental results showed that the average accuracy of the proposed model reached 96.12%, the FPS on RTX 3090 was 172, and the speed on the embedded Jetson TX2 was 20 frames per second. Compared with the original YOLOv5 model, the mAP was improved by 2.21%, the calculation amount was reduced by 26%, and the speed was increased by two times. Conclusion: YOLO-FFD can meet the requirement of identifying fruit varieties and freshness, and improve the falsely detection and missing detection in complex scenes.