PRO-Step：面向检索增强生成的步骤级过程奖励优化
原标题：PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation
AI 摘要
PRO-Step 提出了一种面向检索增强生成（RAG）的步骤级过程奖励优化框架，通过训练生成式过程奖励模型（PRM）同时评估每一步的逻辑有效性和证据支撑，并利用 PRM 引导的价值树搜索构建偏好对，结合步骤级直接偏好优化（DPO）来训练策略模型。实验表明，该方法在五个单跳和多跳问答基准上取得了最佳的平均 EM 和 F1 分数，代码、模型和训练数据已公开。
正文节选
PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation Abstract Retrieval-Augmented Generation enhances Large Language Models by grounding responses in external knowledge, but multi-hop reasoning remains vulnerable to error propagation, where early retrieval failures confound subsequent steps. Standard outcome-based optimization only rewards the final answer, leaving intermediate retrieval and reasoning errors undetected. While existing process-based methods introduc