This study investigates the application of distillation models for the preservation of cultural heritage, with a specific emphasis on traditional Vietnamese instruments (TVI). We systematically evaluate various distillation approaches, including combinations of advanced Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), assessing their performance in terms of model compactness and accuracy. The central aim is to identify a lightweight model that retains high accuracy, ensuring its practical viability in real-world scenarios involving TVI. Our in-depth analysis demonstrates that certain distillation models achieve substantial reductions in computational complexity while preserving the essential classification capabilities crucial for cultural heritage preservation. Notably, the accuracy of these models exceeds 97%, with several combinations reducing model size by approximately 20MB. The detailed evaluation results underscore the potential of these models for efficient management and preservation of heterogeneous cultural datasets.