MASS：基于流形覆盖与稀疏特征覆盖的分层数据选择方法
原标题：Hierarchical Data Selection via Manifold Coverage and Sparse Feature Coverage in LLM Post-training
AI 摘要
该论文提出了一种名为MASS的分层数据选择方法，用于大语言模型的后训练阶段。MASS首先使用密集自编码器学习低维主流形坐标进行粗粒度语义分组，然后利用TopK稀疏自编码器在每个组内进行质量感知的稀疏特征覆盖。在Vision-Flan和LLaVA-CoT上的实验表明，MASS在多个数据预算下优于现有基线，有时仅用少量数据即可达到或超过全量训练的性能。
正文节选
Hierarchical Data Selection via Manifold Coverage and Sparse Feature Coverage in LLM Post-training Abstract As supervised fine-tuning data continues to scale, selecting high-value subsets from large candidate pools is crucial for reducing training cost and improving model performance. Existing methods often measure diversity directly in the original embedding space, where geometric metrics entangle dominant semantic directions, fine-grained supervision differences, and local noise. We address th