cs.AI提交于 2026-10-27已译

Co-Design Bench：面向智能体模型的多模态协同编码评测

Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

Xilin Wei · Shengyuan Ding · Xinyu Fang · Tiancheng Gu · Yuchi Wang · Zhixun Li · et al.

中文摘要

评估人机协同设计，需要检验编码智能体能否将真实用户的多模态(multimodal)请求转化为可运行的应用修订。Co-Design Bench 是一个无需参考答案的基准，包含从37,463条去重查询中整理的273项任务，来源覆盖26,314次真实用户编码会话。其多模态编辑(Multimodal Editing, ME)与代码库编辑(Repository Editing, RE)两条赛道，分别考察带图像、音频、视频或文档输入的局部编辑，以及协调式代码库修订。任务覆盖扎根理解(Grounded Understanding)、UI转换(UI Transformation)与功能实现(Functional Implementation)三种能力维度，并配有标准化请求、可运行的编辑前应用和2,331条隐藏的专家验收标准。判定再反驳(judge-then-refute)协议利用源码与浏览器证据评分，由独立LLM验证器质疑肯定结论，并在没有标准代码或目标截图时审核视觉与交互声明。在统一智能体运行框架下，10个模型端点中最强者平均通过每项任务69.9%的标准，但仅完整满足26.4%的任务。所有端点在涉及三至四类改动的代码库编辑中，标准完成率均低于一至两类改动。

关键要点

01构建无需标准答案的多模态协同编码基准，涵盖273项真实用户任务
02设置多模态编辑与代码库编辑赛道，考察三种核心能力维度
03采用判定再反驳协议，结合源码、浏览器证据与独立LLM验证
04最强模型平均通过69.9%的验收标准，但仅完整满足26.4%的任务
05代码库编辑涉及三至四类改动时，所有端点的标准完成率均下降

解读

尚无解读。

原始英文摘要

arXiv:2610.35138v1 Announce Type: new Abstract: Evaluating human-AI co-design requires testing whether coding agents can translate real users' multimodal requests into working application revisions. We introduce Co-Design Bench, a reference-free benchmark of 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Its two tracks, Multimodal Editing (ME) and Repository Editing (RE), cover localized edits with optional image, audio, video, or document inputs and coordinated repository revisions. Tasks span three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The tasks pair normalized requests and runnable pre-edit applications with 2,331 hidden expert-authored acceptance criteria. Our judge-then-refute protocol grades criteria using source and browser evidence, challenges positive verdicts with a separate LLM verifier, and audits visual and interaction claims without gold code or target screenshots. Under a shared agentic harness, the strongest of ten model endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two. Co-Design Bench provides a testbed for studying the gap between partial requirement satisfaction and complete task success under agentic execution for real-user application revisions.