A significant environmental problem in Vietnam and around the world is litter and garbage left on the street. The illegal dumping of garbage can have negative consequences on the environment and the quality of human life, as can all forms of pollution, in addition to having a large financial impact on communities. Thus, in order to reduce the impact, we require an automatic litter detecting method. In this study, we propose a new method for spotting illegal trash disposal in surveillance footage captured by real-world cameras. The illegal littering is recognized by a deep neural network. An ordered series of frames makes up a video. The order of the frames carries the temporal information, and each frame contains spatial information. To model both of these features, convolutional layers are used for spatial processing, and instead of using recurrent layers for obtaining temporal information, we take advantage of transformer encoder for encoding sequential data by evaluating each element in the sequence’s relevance. Besides, CNNs is also used to extract important features from images, hence lowering input size without compromising performance. This leads to reduce computational requirements for the transformer. Through testing on actual recordings of various dumping operations, we show that the proposed strategy is effective. Specifically, the validation outcomes from the testing data’s prediction reveal an accurate value of 92.5%, and this solution can be implemented in a real-time monitoring system.