In this paper, we introduce an approach to improve performance of Multi-Label Classification of X-Ray images with Self-Supervised Learning (MLCXR-SSL). The SwinT-Compact architecture is also proposed to reduce model complexity and increase computational efficiency. By leveraging contrastive learning, features/representations are extracted from a wealth of unlabeled data, thereby improving data efficiency and overcoming the challenges posed by the restricted labeling of medical data. Our contribution includes refining an architecture to effectively apply self-supervised learning (SSL) to multi-label classification problems. Additionally, we conduct extensive experiments to compare the fine-tuning of the linear classifier from the ImageNet pre-trained model with those from the unlabeled X-ray image pre-trained model. This comparison is performed on both the SwinT architecture and our proposed SwinT-Compact architecture, both based on the Swin Transformer, using the Chest X-ray 14 dataset. Results show a significant performance gain achieved by fine-tuning the unlabeled X-ray image pre-trained model compared to the ImageNet pre-trained model, especially notable in SwinT (AUC 0.809 vs. 0.77). Furthermore, our proposed architecture maintains multi-label classification performance comparable to the SwinT architecture while reducing model complexity and training time.