HakemBench：土耳其语类型化决策基准发布
原标题：HakemBench: A Turkish Benchmark of Typed Decisions
AI 摘要
研究者发布 HakemBench，一个土耳其语「类型化决策」基准，包含 2,346 个条目、4,275 道选择题/是非题/评分题，覆盖事实核查分流、教育、护栏、法律路由、内容审核、垃圾邮件与钓鱼、客户支持七个赛道，采用 CC BY 4.0 完全开放。其评分工具同时衡量决策质量（宏 F1）、校准度（归一化 Brier 分数）与选择性自动化（归一化广义风险-覆盖率曲线下面积），并以几何平均合成。多数金标签由单一 AI 模型家族的盲测结果与其他模型家族大模型投票对比生成，未经人工验证；16 行榜单中榜首综合分 0.888，而该实验室自家模型排第 7（0.660），且其护栏、审核与客服分数因训练数据受早期测试结果影响而被标记。
正文节选
HakemBench: A Turkish Benchmark of Typed Decisions Abstract HakemBench is a Turkish benchmark of typed decisions, in which the model under test reads a text, a question and a fixed set of options and returns a probability for every option. Version 1.0 is released fully open under CC BY 4.0, with 2,346 items and 4,275 choice, yes/no and score questions in seven tracks (fact-check triage, education, guardrails, legal routing, moderation, spam and phishing, and customer support). One harness scores