Grain discoloration disease of rice is an emerging threat to rice harvest in Vietnam as well as all over the world and it acquires specific attention as it results in qualitative loss of harvested crop. An accurate classification is preliminary to any kind of intervention. Unfortunately, collecting enough grain discoloration data as well as building and training a machine learning model from scratch is next to impossible due to the lack of hardware infrastructure and finance support. It painfully restricts the needs of rapid solutions to deal with the disease. For this purpose, this paper exploits the idea of transfer learning which is the improvement of learning in a new prediction task through the transfer of knowledge from a related prediction task that has already been learned. By utilizing convolutional neural networks trained with our collected data, our experiment shows that the proposed idea performs perfectly and achieves the classification accuracy of 88.2% with the acceptable training time on a normal laptop.