English as a foreign language (EFL) students’ lack of engagement directly affects their learning outcomes, and it has captured the attention of many researchers. Factors affecting EFL students’ participation may be differed by student and lecturer perceptions, which will cause differences in these stakeholders’ solutions to classroom difficulties. Accordingly, this study measured Non-English major (NEM) students and EFL lecturers’ perceptions of factors hindering students’ participation in English-speaking classes. About 156 NEM freshmen and 14 lecturers responded to a 35-item questionnaire containing five primary clusters: linguistic, cognitive, affective, pedagogical, and social-cultural factors. The study further employed semi-structured interviews with six students and four lecturers. The results found some significant differences between lecturer and student perceptions. On the one hand, the students did not consider the large class size, insufficient time for in-class practice, and students’ tendency to remain silent as hindering their class participation. On the other hand, the lecturers considered vital barriers: students’ insufficient proficiency, teachers’ poor lessons, or teacher-student relationship. Based on the findings, teacher professional development activities are expected to be one of the solutions to EFL students’ insufficient engagement in English-speaking classes.