In this paper, we propose the random forest algorithm of biggest margin trees RF-BMT for the multi-class classification. The novel algorithm enhances the classification of chest X-ray images, specifically for distinguishing between normal, covid-19, edema, mass-nodule, and pneumothorax cases. Our approach combines contrastive learning with our proposed algorithm to improve performance and address the limitation of labeled data by leveraging a large amount of unlabeled data for learning features. We propose training the algorithm on the features extracted from the linear fine-tuned model of Momentum Contrast (MoCo), which is trained on Resnet50 architecture. The RF-BMT algorithm plays a role as a replacement for softmax in deep networks. Based on the empirical results, our proposed RF-BMT algorithm demonstrates substantial improvement compared to solely fine-tuning the linear layer both the ImageNet pretrained model and the MoCo pretrained model, reaching an impressive accuracy rate of 88.4%.