To realize the rapid detection of normal peppers and defective peppers (dark spots, insect marks and stripe skin), image acquisition and median filtering were used for denoising. 24 color feature variables and 20 texture feature variables of sample images were extracted. Threshold segmentation in the Otsu algorithm was used to extract two morphological feature variables. The Successive Projections Algorithm (SPA) was used to optimize 14 feature variables. Modeling was done based on the Least Squares Support Vector Machine (LS-SVM) method. The detection result of SPA-LS-SVM model is 98.67%, which is the best.