NCP-ArchPreview：通过下一概念预测迈向潜在空间语言模型
原标题：Paper page - NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
AI 摘要
该论文提出 NCP-ArchPreview，一种在自回归预训练中同时进行下一 token 预测（NTP）和下一概念预测（NCP）的潜在空间语言模型。模型通过从隐藏状态构建乘积量化概念词表，并用专门的 Concept Module 预测跨多个 token 的离散概念，再反馈到 token 层指导生成。模型规模达 8.9B 参数，在 Dolma-3 的 5.73T token 上训练，仅用 51.3% 训练 token 即达到 OLMo-3-7B 的最终预训练损失，下游宏平均高出 2.45 分，GSM8K 提升 5.99 分。
正文节选
NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction Abstract NCP-ArchPreview is a large latent-space language model that jointly trains next-token and next-concept prediction to improve pretraining efficiency and downstream performance. We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Predicti