学习评审清单以减轻科学同行评审中的抽签效应
Learning Checklists to Reduce Lottery Effects in Scientific Peer Review
Xuanyou Liu · Ziyang Guo · Jason Hartline · Jessica Hullman
中文摘要
同行评审中的抽签误差是指少数受邀评审只关注论文的部分问题，而更广泛的合格评审者本会提出更多问题。利用从历史评审语料中学习到的批评意见清单，可以减轻此类误差。该方法生成一棵清单树，表示原子化评审主张空间的层次化粗化，并选择使评审者间相关一致性达到理论最优度的粗化版本。研究通过 LLM 辅助流程实现该方法，并在 NeurIPS 2021 与 ICLR 2022 评审数据上测试。在采用 LLM 评审者的模拟评审中，学习所得清单使三人委员会评审的均方误差(MSE)在 NeurIPS 上降低 58%，在 ICLR 上降低 21%；两个会议中，相关一致性最高的清单，在三人及以上委员会中均取得最低误差。在包含 20 名参与者的人类研究中，以 NeurIPS 2021 排序为基准，学习所得的 NeurIPS 清单将论文排序的 MSE 从 0.48 降至 0.22。
关键要点
- 01针对少数评审遗漏其他评审者会提出的问题，定义同行评审抽签误差。
- 02从历史评审语料生成批评清单树，并优化评审者间的相关一致性。
- 03在 LLM 模拟评审中，三人委员会 MSE 在 NeurIPS 降低 58%、ICLR 降低 21%。
- 0420 人实验表明，学习所得清单将排序 MSE 从 0.48 降至 0.22。
解读
尚无解读。
原始英文摘要
arXiv:2610.28666v1 Announce Type: cross Abstract: Reviewer lottery error describes error due to a small set of peer reviews attending to only some of the issues with a paper that a larger pool of eligible reviewers would raise. We show this error can be reduced by surfacing a checklist of criticisms learned from a historical review corpus. Our approach generates a tree of checklists representing coarsenings of the atomic review claim space, then selects the coarsening that maximizes a theoretically-motivated measure of correlated agreement among reviewers. We realize this approach with an LLM-assisted pipeline and test it on NeurIPS 2021 and ICLR 2022 review data. In simulated review with LLM reviewers, the learned checklist reduces the mean squared error (MSE) of a three reviewer committee by 58% on NeurIPS and 21% on ICLR, and on both venues the checklist with the highest correlated agreement has the lowest error for committees of three or more. In a human study with 20 participants, the learned NeurIPS checklist lowers the MSE of a paper ranking, measured against the NeurIPS 2021 order, from 0.48 to 0.22.