CompLit：科学文献搜索基准必须涵盖隐含、累积与未满足需求
CompLit: Scientific Literature Search Benchmarks Must Cover Implicit, Cumulative, and Unmet Needs
Dayoon Ko · Jihyuk Kim · Soyeong Jeong · Young-jun Lee · Dahyun Lee · Juyeon Kim · et al.
中文摘要
arXiv:2610.28658v1 Announce Type: cross Abstract: 我们有多大把握相信智能体能找到所需论文？研究人员通常从已读文献出发，逐步搜索尚未接触的内容，而需求可能仅被论文隐含表达，也会随阅读不断累积，甚至存在于尚无论文覆盖的空白区域。CompLit 覆盖 arXiv 全部八个领域，并分别设置一种诊断场景以测试上述需求。最强智能体 Codex (GPT-5.6-Sol) 与 Claude Code (Opus 4.8) 可回答 86–87% 的 Standard 查询，但面对隐含需求时降至 66–68%；需求累积后，两者又分别下降 6 和 15 个百分点；对没有论文满足的查询，最多有 53% 选择放弃。没有智能体能在所有领域领先，且对 2019 年以前或 2023 年以后发表的论文准确率均较低。追踪其搜索过程发现，智能体经常找到正确或最接近的论文，却误判其相关性。提升 CompLit 表现，是迈向可信地找到所需论文或明确告知其不存在的一步。
关键要点
- 01CompLit 覆盖 arXiv 八个领域，诊断隐含、累积与未满足需求。
- 02Codex (GPT-5.6-Sol) 与 Claude Code (Opus 4.8) 的 Standard 查询答对率为 86–87%。
- 03隐含需求下答对率降至 66–68%，累积需求后分别再降 6 和 15 个百分点。
- 04面对无论文满足的查询，智能体最多仅对 53% 作出弃答。
- 05各智能体均无全领域领先表现，对 2019 年前或 2023 年后论文准确率较低。
解读
尚无解读。
原始英文摘要
arXiv:2610.28658v1 Announce Type: cross Abstract: How much can we trust an agent to find the paper we need? As researchers, we search the literature from what we have read toward what we have not. We thus describe what we want in our own terms, which a paper may show only implicitly; our requirements grow as we read; and what we seek may lie in an empty region no paper yet covers. We introduce CompLit, a literature search benchmark spanning all eight arXiv domains, with one diagnostic setting for each of these needs. The strongest agents, Codex (GPT-5.6-Sol) and Claude Code (Opus 4.8), answer 86-87% of Standard queries, yet fall to 66-68% when requirements are implicit (implicit), lose another 6 and 15pp, respectively, as they accumulate (cumulative), and abstain on at most 53% of the queries no paper satisfies (unmet). No agent leads in every domain, and all are less accurate on papers from before 2019 or after 2023. Tracing their search, we find that agents often reach the right paper, or the closest one, yet misjudge it. Doing well on CompLit is a step toward agents that we can trust to find the paper we need, or to tell us that none exists.