[Objective] Aiming at the problems of low accuracy and poor robustness in the previous classification methods based on artificial features. [Methods] A strong generalization automatic classification method of potato shape and size was proposed in this study. First, two potato ViT models were built based on Transformer model to complete potato shape grading and size grading tasks in parallel. Secondly, a robust model was trained by using migration strategy and data augmentation method. Finally, the effectiveness of this method in potato grading was verified by quantitative analysis of test sets. [Results] The experimental results show that the potato ViT model achieves 96.36% and 94.75% for potato shape classification, and 89.66% and 85.16% for size grading in terms of accuracy and μ_F1 index. The classification accuracy was better than VGG16, ResNet50 and MobileNetV3 network models. [Conclusion] The results shows that it is feasible to apply the proposed method for the real-time and accurate detection of potato shape and size. The results of this study can provide theoretical and technical support for potato intelligent grading.