There are the diseased fruits left in the product of the automatic fresh chili destemming system. This will reduce the processing quality, so it is necessary to develop a grading system. This research focused on building a model to identify the damages caused by diseases on fresh destemmed chili fruits. A convolution neural network (CNN) model was built and trained with the support of OpenCV, TensorFlow, and Keras libraries in order to identify damages on the fruit body. An independent test was performed on 184 fruits with 1840 identify times and achieved an accuracy rate of 90.8%. This result was acceptable and would be used as a basis for the future classification system. Further improvements could be investigated to achieve a higher success rate.