Pneumonia is one of the most serious diseases for infants and young children, people older than age 65, and people with health problems or weakened immune systems. From numerous studies, scientists have found that a variety of organisms, including bacteria, viruses, and fungi, can be the cause of the disease. Coronavirus pandemic (COVID-2019) which comes from a type of pneumonia has been causing hundreds of thousands of deaths and is still progressing. Machine learning approaches are applied to develop models for medicine but they still work as a black-box are difficult to interpret output generated by machine learning models. In this study, we propose a method for image-based diagnosis for Pneumonia leveraging deep learning techniques and interpretability of explanation models such as Local Interpretable Model-agnostic Explanations and Saliency maps. We experiment on a variety of sizes and Convolutional neural network architecture to evaluate the efficiency of the proposed method on the set of Chest x-ray images. The work is expected to provide an approach to distinguish between healthy individuals and patients who are affected by Pneumonia as well as differentiate between viral Pneumonia and bacteria Pneumonia by providing signals supporting image-based disease diagnosis approaches.