审计LLM量子代码生成评测：基准饱和与评测器缺陷
原标题：Benchmarking the Quantum Leap: Auditing LLM Evaluation for Quantum Code Generation
AI 摘要
该论文对量子代码生成领域的LLM评测进行了系统审计，在五个包含530个任务的基准上评估了15个模型，并提出了新的96任务基准QuantumBenchEval。研究发现QiskitHumanEval-Standard对最强模型已接近饱和（98.7% pass@1），而评测设计缺陷影响巨大：移除提示未指定的检查后，GPT-6 Astra的pass@1从0%升至94.1%。审计还发现QuanBench评测器存在未执行单元测试、参考函数泄漏等问题，表明排行榜分数不足以反映真实进展。
正文节选
Benchmarking the Quantum Leap: Auditing LLM Evaluation for Quantum Code Generation Abstract Large language models (LLMs) are increasingly capable of generating quantum programs, but reliable evaluation remains difficult: benchmarks differ in task composition, scoring procedures, and failure handling, while leaderboard rankings can obscure uncertainty and evaluator defects. We evaluate 15 hosted frontier, open-weight, quantum-specialized, and retrieval-augmented models across five primary quantum