PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix
Marco Paina · Mark Niklas M\"uller · Niels M\"undler-Sasahara · Martin Vechev
主动 Bug 发现是软件工程的关键问题，但编码智能体在该领域的能力研究不足，特别是跨整个仓库发现所有人类维护者会修复的 Bug 这一任务尚未被研究。为此，PROBE 基准被提出，包含 20 个多样化 GitHub 仓库在 18 个月内修复的 2,457 个 Bug，并提供了将候选发现与该部分参考集匹配的方法。评估显示，前沿模型编码智能体相对于参考集的召回率均低于 21%，精确率低于 9%。虽然更强模型将召回率从 8% 提升至 17%，但精确率未改善。令人惊讶的是，尽管精确率低，LLM 和人类评估者仍判定 85% 的发现有效。关键发现是，不同智能体发现的 Bug 分布相似，但与维护者修复的 Bug 分布存在显著差异。鉴于 Bug 发现的实用价值及前沿模型的低召回率，该基准对模型用户和开发者均具有重要价值。
尚无解读。
arXiv:2610.35987v1 Announce Type: cross Abstract: Proactive bug discovery is an important software-engineering problem. Even though coding agents are already widely deployed in this domain, their capabilities remain understudied. In particular, the challenging task of finding all bugs that human maintainers would repair across an entire repository has not been studied yet. To close this gap, we introduce PROBE, a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories, together with a method for matching candidate findings to this partial reference set. We evaluate coding agents across frontier models and find they all achieve a recall of below $21\%$ and a precision of less than $9\%$ with respect to the reference set. While stronger models increase recall substantially from $8\%$ to $17\%$, they do not improve precision. Surprisingly, despite the low precision, LLM and human evaluators still judge $85\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers. The practical relevance of bug discovery and the low recall of frontier models make this benchmark a valuable resource for both model users and developers.
``` # Code Review 规范 你是 golang 编程的专家，现在对代码进行 Code Review，请按照以下步骤进行: （若遇到 git 命令分页的情况，请加上`| cat`） ## 步骤一：代码规范检查 - [ ] 是否遵循最佳实践 @.cursor/rules/rule-code-spec.mdc - [ ] 是否存在重复代码 - [ ] 是否有充分的注释和文档 - [ ] 是否易于理解和维护 ## 步骤二：功能问题分析 - [ ] 分析本次代码改动的影响…
### 1. 规划与分阶段 将复杂的工作分解为 3-5 个阶段。在 `IMPLEMENTATION_PLAN.md` 中记录： ```text ## 阶段 N: [Name] **目标**: [具体可交付成果] **成功标准**: [可测试的结果] **测试**: [具体的测试用例] **状态**: [未开始|进行中|已完成] ``` * 随着进展更新状态 * 所有阶段完成后删除文件 ### 2. 实现流程 1. **理解** - 研究代码库中的现有模式 2. **测试**…
This is the job of the model Create a test pinch and Raman hotel minis verification test in Raymond for SBI from the scratch along with the design as well dispatched be in the uvm environment and also it is pinch is should be expendable wh…
I’m tired of using Claude Code to build my code because of tokens limits can Ollama build code scripts agentic workflow?
Please debug the current codebase and implement a robust development plan to ensure the system is fully functional. Prioritize refactoring the application architecture to follow industry best practices, resolve existing build errors, and en…
Role: Principal AI Project Manager, Senior Prompt Engineer, and Multi-Agent Workflow Orchestrator. Context: Continue the existing project. Inspect and follow all current project rules, architecture decisions, governance requirements, envir…