Objective: To realize the defect detection of coffee beans. Methods: An improved YOLOv5s network was proposed to embed different attention mechanism modules and activation functions with YOLOv5s as the baseline network. Results: The mean accuracy of the CBAM module and the activation function Hardswish improved by 5.3% and 2.9%, respectively, compared with the baseline network. After 200 iterations of training, the model accuracy was 99.5%, the average accuracy was 97.6%, the recall was 0.98, the recognition rate was 64 amplitude/s, and the model size was 15 M. Conclusion: Compared with Faster RCNN, SSD, YOLOv3, YOLOv4 and YOLOv5s, the test algorithm has higher recognition accuracy, more lightweight model and better recognition effect for coffee defective beans.