Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
Journal:
arXiv
Published Date:
Apr 20, 2025
Abstract
The exponential growth of Large Language Models (LLMs) continues to highlight
the need for efficient strategies to meet ever-expanding computational and data
demands. This survey provides a comprehensive analysis of two complementary
paradigms: Knowledge Distillation (KD) and Dataset Distillation (DD), both
aimed at compressing LLMs while preserving their advanced reasoning
capabilities and linguistic diversity. We first examine key methodologies in
KD, such as task-specific alignment, rationale-based training, and
multi-teacher frameworks, alongside DD techniques that synthesize compact,
high-impact datasets through optimization-based gradient matching, latent space
regularization, and generative synthesis. Building on these foundations, we
explore how integrating KD and DD can produce more effective and scalable
compression strategies. Together, these approaches address persistent
challenges in model scalability, architectural heterogeneity, and the
preservation of emergent LLM abilities. We further highlight applications
across domains such as healthcare and education, where distillation enables
efficient deployment without sacrificing performance. Despite substantial
progress, open challenges remain in preserving emergent reasoning and
linguistic diversity, enabling efficient adaptation to continually evolving
teacher models and datasets, and establishing comprehensive evaluation
protocols. By synthesizing methodological innovations, theoretical foundations,
and practical insights, our survey charts a path toward sustainable,
resource-efficient LLMs through the tighter integration of KD and DD
principles.
