Data-DPO：面向目标模型数据选择的直接偏好优化方法
原标题：Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training
AI 摘要
Data-DPO 是一种面向目标模型的监督微调数据选择方法，通过单步探测观察目标模型在不同样本上的训练反馈，将样本间的激活差异转化为成对数据偏好，并训练轻量级奖励模型来学习目标模型感知的数据偏好。在最终选择阶段，Data-DPO 结合目标模型偏好、外部质量评分和边际多样性来构建训练子集。在 Vision-Flan 和 LLaVA-CoT 数据集上的实验表明，Data-DPO 在多种数据预算下均优于现有数据选择基线，并稳定超过全量数据训练的性能。
正文节选
Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training Abstract Data selection in supervised fine-tuning aims to select a small set of effective samples from large-scale candidate data, reducing training cost while preserving model performance. However, existing methods usually treat data value as a relatively static property, and pay limited attention to the compatibility between data and the capability distribution of the target model. To address this iss