RealCLI：面向 CLI 智能体人机协作的统一评估基准
原标题：RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents
AI 摘要
论文提出 RealCLI，一个配对基准，用于在真实 CLI 任务上统一评估任务执行效果与人机协作质量。它构建了五维协作框架（可理解性、可引导性、观察与恢复、信任校准、安全与风险意识），细化为 22 个子维度和 51 个交互场景模板，并提供两种模式：RealCLI-EXEC 预先给出完整规格，RealCLI-COLLAB 由脚本化用户逐步提供信息、纠正和授权决策。评估显示前沿智能体最佳配置平均分仅 49.2%，联合协作得分峰值仅 37.6%，说明任务有效性与协作质量存在明显差距，模型在可引导性、信任校准和安全边界管理上持续表现不佳。
正文节选
RealCLI: Toward a Unified Evaluation of Human–Agent Collaboration for CLI Agents Abstract Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks.