To make the cherry defect detection and identification system meet the real-time requirements, it was proposed to use the convolutional neural network model as the basis and used the SDSoC development platform to complete the design of FPGA for rapid detection and identification of cherry defects. By optimizing data transmission, multiplexing the general matrix multiplication function (GEMM) in the network model and parallelizing the design of the convolution operation, PL-side hardware acceleration was realized. Using the SDSoC platform, a high-level language mapping convolutional neural network model was used on the PS-side, which saved a lot of development time while achieving the required performance. The results showed that, compared with the pure software method, the speed based on the Zynq7020 hardware development platform had increased by more than 2.19 times. Compared with the CPU platform, the speed was almost the same.