Destemming fresh chilli fruit (capsicum) in large productivity is necessary, especially in the Mekong Delta region. Several studies have been done to solve this problem with high applicability, but a certain percentage of the output consisted of cracked fruits, thus reducing the quality of the system. The manual sorting results in high costs, and low quality, so it is necessary that automatic grading is performed after destemming. This research focused on developing a method to identify and classify cracked chilli fruits caused by the destemming process. The convolution neural network model (CNN) was built and trained to identify cracks; then appropriate control signals were sent to the actuator for classification. Image processing operations are supported by the OpenCV library, while the TensorFlow data structure is used as a database and the Keras application programming interface supports the construction and training of neural network models. Experiments were carried out both in the static and working conditions, and respectively achieved an accurate identification rate of 97 and 95.3%. In addition, a success rate of 93% was found even in case the chilli body is wrinkled due to drying after storage time at 120 hours. Practical results demonstrate the reliability of the model was useful and acceptable.