Physical Education Subject (PES) is a challenging problem for conveying IT into education. From the perspective of Artificial Intelligence (AI) and realizing PES’s abstractness in knowledge representation, we pay attention to the idea of an intelligent system with a combination breath of computer vision and speech recognition. Inspired by that vision, we propose a support system called PETSAI (another name: PETSAI-Ext) for teaching the movements of the PES. Our primary idea is to introduce a “audio-visual” framework to compute the scores of the movements. To this end, we leverage human pose estimation (HPE) to detect a student’s skeleton for each movement and utilize convolutional neural networks (CNN) for motion and gesture classification. In addition, we also use the power of speech recognition to control the entire framework. Two operations, the speech and image processes, will be performed in parallel. Moreover, we also provide an algorithm to calculate scores (from gestures to movements). In addition, the regression models has also been introduced to calculate scores in this paper. Finally, we implement the system on both platforms, including the website and desktop application.