高效 LLM 知识蒸馏：离线 Top-K Logits 与融合分块 KL 损失
原标题：Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
AI 摘要
Multiverse Computing 的研究人员提出了一种高效的 LLM 知识蒸馏方法，通过离线缓存教师模型的 Top-K logits 和融合分块 KL 损失，在保持蒸馏质量的同时，将训练速度提升约 29%，吞吐量最高提升 41%，并支持在单 GPU 上训练 4 倍上下文长度（32,768 tokens）。该方法减少了峰值内存占用，使大规模蒸馏和数百次消融实验变得可行，相关实现已在 GitHub 上开源。
正文节选
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss Abstract Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD). This recovery step largely decides the final quality, yet it is expensive. We present a practitioner's study of how to make distillation training efficient, orga