Rice is a staple food and a gluten-free grain. According to a survey on agriculture and natural resources with international plant pathology by the University of California, rice leaf diseases highly affect rice production loss. Scientists, farmers, and the management team are always on the hunt for a method for early detection and classification of rice leaf diseases. Such a method will facilitate accurate and prompt treatment of diseases and will prevent their spread. Deep learning techniques provide an edge over the manual way of classification in many ways. Our work uses a deep learning-based approach for rice leaf disease classification. MobileNetV2 is used along with transfer learning for classification. The main purpose of this research is to utilize a challenging dataset that is not carefully selected and exploit transfer learning techniques. The functional capabilities of the classification model are improved by a reduction in the training time and parameter size optimization. Four different datasets that contained different characteristics were collected. These datasets were then combined to construct a combined dataset of 7,445 images containing five types of rice leaf diseases. The proposed model and another convolutional neural network (CNN) have been used as baseline models on the five datasets and results have been analyzed. According to the experimental analysis, the proposed model has the best classification accuracy of 99.17%. It provides a significant increase in accuracy using fewer parameters as compared to other models. Experimental findings show the feasibility and efficiency of the proposed model.