cs.SE提交于 2026-10-22已译

VeeReview：从 V 模型视角衡量智能体编写拉取请求中的构建—验证配对

VeeReview: Measuring Build-Verify Pairs in Agent-Authored Pull Requests from a V-Model Perspective

Gabriel Dautovi · Jie JW Wu

中文摘要

评估拉取请求需要与所提变更相对应的验证证据，但现有编码智能体基准和拉取请求结果研究未直接考察这种关系。研究检验 V 模型的构建—验证配对能否衡量拉取请求中的验证证据，并支持智能体与人类编写变更之间的比较。VeeReview 利用仓库信号和语言模型评审器，将变更归入四类构建—验证配对；三类由受确定性信号约束的模型判断决定，另一类“模块设计—单元测试”由差异与仓库目录树规则决定。依据两名评分者标注的 25 个拉取请求参考集进行评估后，该方法应用于来自 741 个仓库的 1,454 对智能体与人类拉取请求匹配样本。四类配对中，智能体拉取请求的仅构建率均较低，与预注册假设相反（混合效应优势比为 0.52 至 0.86）；“架构—集成”不显著。控制关联议题、随附测试和变更规模后，匹配率差异缩小 34% 至 60%，规则判定配对的标准化区间包含零。其与裁决标签的一致率为 4% 至 60%，未达到 80% 的预注册校准标准，评分者间一致性也较弱（kappa = 0.154）。规则配对表明，语言模型评判的过程指标会部分反映拉取请求的呈现方式，而非其背后的验证工作。此类指标需包含机械化判定部分和经验证的人类参考，方可用于比较验证实践。

关键要点

01现有基准未直接衡量拉取请求变更与验证证据的关系
02结合仓库信号、语言模型评审器及差异规则判定四类构建—验证配对
03应用于 741 个仓库的 1,454 对智能体与人类拉取请求
04控制议题、测试和规模后，智能体与人类的匹配率差异缩小 34% 至 60%
05一致率未达 80% 标准，且 kappa = 0.154，显示方法校准不足

解读

尚无解读。

原始英文摘要

arXiv:2610.29143v1 Announce Type: cross Abstract: Context: Reviewing a pull request requires verification evidence that addresses the proposed change. Coding agent benchmarks and studies of pull request outcomes do not directly assess this relationship. Objective: We test whether the V-Model's build and verify pairs can measure verification evidence in pull requests and support comparisons between agent-authored and human-authored changes. Method: VeeReview classifies four build and verify pairs using repository signals and a language-model judge. Three pairs use model judgements constrained by deterministic signals, and a rule over the diff and repository tree decides Module Design-Unit Test. We evaluated VeeReview against a 25-pull request reference set labelled by two raters and applied it to 1,454 matched pairs of agent and human pull requests from 741 repositories. Results: VeeReview assigned lower build-only rates to agent-authored pull requests on all four pairs, contrary to the preregistered hypothesis (mixed-effects odds ratios 0.52 to 0.86). Architecture-Integration was not significant. Standardization for linked issues, shipped tests and size reduced matched-rate differences by 34% to 60%, and the rule-decided pair's standardized interval included zero. The instrument did not meet its preregistered calibration criterion: agreement with adjudicated labels was 4% to 60% against a required 80%, and inter-rater agreement was weak (kappa = 0.154). Conclusions: The rule-decided pair shows that a process measure judged by a language model partly reflects how a pull request is presented rather than the verification work behind it. We report the cohort differences as observations made with this instrument, and we argue that such measures need a mechanically decided component and a validated human reference before they can support comparisons of verification practice.