Objective: In order to solve the problem of inaccurate defect extraction caused by complex and variable color and irregular texture changes of tomato surface defects, the defect segmentation method based on image local variance with brightness correction was proposed. Methods: On the basis of using histogram threshold segmentation method to segment calyx and stem scar and the method of domain pixel weighted sum to replace the original pixel to complete the brightness correction, the gray image of tomato surface was divided into several image blocks, and the color of each block was characterized by image pixel variance, and then the defect and healthy area were separated. SVM model was used to detect the proportion of tomato surface defect area in the original tomato area. Results: Considering the brightness correction, the accuracy of tomato defect area extraction could be improved by 27.74%. On this basis, compared with the global threshold, dynamic threshold and regional growth algorithm, the defect extraction method based on image local variance could accurately achieve the quasi-deterministic and complete extraction of tomato surface defects, and the accuracy of the Gauss-SVM model with the defect area ratio as the input for tomato surface defect detection reached 96%. Conclusion: Considering brightness correction, the SVM defect extraction method based on image local variance is suitable for tomato surface defect detection.