Assembly line detection more depend on the machine vision technology, and image segmentation is the key step in the detection. To solve the problem of slow speed and low accuracy of defects segmentation methods to the surface defects of food packing, proposed a segmenting method based on image difference and mathematical morphology in OpenCV. Firstly, compute the template image of the target image and filter them by a low pass filter. Secondly, differential operation is happened on the two pretreated images to obtain the differential image. Finally, using morphological opening operation to remove the noise on the difference image and get the position of defects image. In this paper, 30 groups of defect samples, such as stain, misting and stripping, were selected and the experimental results were recorded. The results showed that the average accuracies of the proposed method were 91.51%, 93.41%, 94.14%, respectively, and the average segmentation time was only 46.6 ms.