Mangoes are among the tropical fruits that provide high export value. Therefore, grading them based on physical and external features is required to satisfy the exporting criteria. An image acquisition system has recently been developed to capture the mango’s 360∘ views. This study developed a computer vision system (CVS) to extend this system towards automatic grade classification of mango fruits based on the mango’s weight and imperfect skin area. The front-view images of four major faces could be identified with a higher accuracy of 97.98% by applying an average filter on the skin area. The mango region in these major-face images was then detected so that the skin defects could measured, and the mango’s 3D dimensions and area in pixel numbers could be calculated to estimate the mango’s weight with an average error of about 2.5% using a ridge regression model. Experimental results with 120 identified major-face images with artificial blemishes showed an average percentage error of 7.28%. At the controlled conveyor speed of 41mm/s for favorable image capture, the system could sequentially analyze about 1452 mangoes per hour for blemish area calculation. With such high throughput, the proposed CVS could be tailored to build an automatic mango grading system based on the mango’s weight and skin imperfections.