HalluPeer:面向科学同行评审中幻觉检测的分类法驱动基准
HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviews
Tzu-Ling Lin · Dong-Ting Yao · Teng-Fang Hsiao · Wei-Chih Chen · Hong-Han Shuai
中文摘要
同行评审规模持续扩大,推动大语言模型(LLMs)作为评审助手的使用,但 LLMs 会生成流畅却缺乏依据的主张,削弱评审可靠性。现有幻觉评测基准并非针对同行评审设计,而同行评审中的事实核查需要将主张与篇幅长且技术性强的论文内容对齐。HalluPeer 是一个用于检测科学同行评审中幻觉的基准,提供论文内容、人工撰写的评审以及注入幻觉后的评审三组对齐数据,并标注用于检测、分类与定位。其流水线构建了同行评审特有的幻觉分类法,识别评审上下文,并以自动化过滤方式注入幻觉。在 12K 篇论文与 38K 条评审上的实验表明,现有检测器难以将幻觉与正当批评区分开来;在真实评审上的评估进一步证明 HalluPeer 所定义的幻觉模式确实出现在实际同行评审中,凸显了面向来源的核查的迫切需求。项目页面见 https://github.com/Lin-TzuLing/HalluPeer.git
关键要点
- 01问题:LLM 用于同行评审时会产生缺乏依据的主张,而现有幻觉评测基准未针对评审场景设计
- 02方法:构建 HalluPeer 基准,提供论文、人工评审、注入幻觉的评审三组对齐数据,基于领域分类法识别评审上下文并自动化注入幻觉
- 03结果:在 12K 篇论文与 38K 条评审上的实验显示,现有检测器难以区分幻觉与正当批评
- 04验证:在真实评审上的评估表明 HalluPeer 定义的幻觉模式在实际同行评审中确实存在
- 05意义:凸显同行评审场景下进行来源感知核查的迫切需求
解读
尚无解读。
原始英文摘要
arXiv:2609.03580v2 Announce Type: replace-cross Abstract: The growing scale of academic peer review has motivated the use of Large Language Models (LLMs) as review assistants, yet LLMs can generate fluent but unsupported claims that undermine review reliability. Existing hallucination benchmarks are not designed for peer review, where verification requires grounding claims in long, technical papers. We introduce HalluPeer, a benchmark for detecting hallucinations in scientific peer reviews, providing aligned triples of paper content, human-written reviews, and hallucination-injected reviews, annotated for detection, classification, and localization. Our pipeline induces a peer-review-specific hallucination taxonomy, identifies review contexts, and injects hallucinations with automated filtering. Experiments on 12K papers and 38K reviews show that existing detectors struggle to separate hallucinations from legitimate critique, while evaluation on authentic reviews demonstrates that HalluPeer-defined hallucination patterns occur in real peer reviews, highlighting the critical need for source-aware verification. Our project page can be found in https://github.com/Lin-TzuLing/HalluPeer.git