Objective: Taking raw potatoes as the objects, an automatic sorting system was designed to realize the automatic identification of potato under the set standard, which provided technical support for the processing and production of potato products. Methods: The control flows of sorting system and sorting algorithms were constructed. Images of two sides of a potato were acquired automatically through automatic transmission, machine vision acquisition and suction pressure turning. The image restoration algorithms were used to eliminate motion blur and the detection algorithms of area ratio, length diameter and bulge were designed to detect potatoes deformity, germination and size of potatoes. A neural network model was established based on color features to classify green skin, discoloration and normal color of potatoes. Results: The BP neural network algorithm was used to predict the appearance color class of green skin, disfigured spots and normal. The average accuracy of prediction classification of neural network is 96.2% by measuring the prediction model with error score. The sorting system was tested by selecting mixed samples. Referring to the sorting standard, the identification accuracy of potatoes reached 95.92% and the processing time of a single potato is 3.76 s. The system runs stably. Conclusion: The method is feasible for precise sorting of raw potato as processing materials, which meets the needs of sorting potatoes in the front end of processing line.