External features such as color, size, weight, and defect area are among the critical parameters for grading mangoes for export purposes. This paper describes a novel computer vision system (CVS) for the accurate measurement of mango skin defects. A novel algorithm called mango skin image slicing (MSIS) was developed to generate mango slices, each of which includes points of pseudo-equidistance from the camera to accurately calculate the defect area. Based on images captured from 360-degree views, defects on the entire mango surface can be detected. For the first time, a strategy for preventing multiple considerations of the same defects from 360-degree images was introduced based on MSIS. The experimental results revealed that recalculations of the same defects were avoided, and negligible minor defects were excluded, complying with the guidance of the mango exporting firms. The measurement accuracy was evaluated with artificial defects of 100 mm2, yielding a mean error of 6.0 ± 1.4 mm2, which was suitable for defect area measurement considering the mango grading standard. The CVS throughput of approximately 644 mangoes per hour was also a reasonable trade-off for the achieved accuracy and great potential of extending the CVS for more sophisticated classification of non-negligible defects based on the advantages of mango images captured at closer camera–object distances.