The application of Artificial Intelligence (AI) in medicine has been a leading concern worldwide. Artificial intelligence-based systems not only support storing a large amount of data but also assist doctors in making a diagnosis. In addition, deep learning has obtained numerous achievements that greatly supported the development of image-based diagnostic methods. On the other hand, deep learning models still work as a black box that makes interpreting the output a challenge. Diagnosis based on images is currently a trend that plays a key role in clinical treatment by discovering abnormal regions for disease diagnosis. This paper proposes a computer-aid diagnosis system to support a pleural effusion diagnosis based on Chest X-ray (CXR) images. This study investigated several shallow convolutional neural network architectures that classify CXR images as well as the technique for processing imbalanced data using oversampling technology. The best model in the experiments was chosen to generate explanations using the Local Interpretable Model-agnostic Explanations (LIME) to support providing signals for pleural effusion diagnosis. The proposed method is expected to provide more informative CXR images of the pleural effusion diagnosis process.