DVBench:面向真实世界 UVM 设计验证智能体的端到端基准
DVBench: End-to-End Benchmark for Real-World UVM Design Verification Agents
Jiaru Qian · Jiawei Li · Xiaolong Hu · Zheng Shen · LiMing Zhao · Xin Zhang · et al.
中文摘要
设计验证(DV)仍是硬件开发的主要瓶颈,使用通用验证方法学(UVM)进行大规模验证高度依赖人力。大语言模型(LLM)与智能体系统的最新进展为 DV 自动化带来新机会,但现有基准主要构造彼此解耦的子任务,偏离真实工作流。DVBench 是首个面向真实世界 UVM 设计验证智能体的端到端基准,包含 9 个功能域、5 个难度等级的 95 个任务,覆盖多样验证场景与设计规模。每个任务要求智能体独立完成完整验证流程。DVBench 通过编译、覆盖率和缺陷检测评估最终测试平台,同时评估特征列表、测试计划和参考模型等中间产物。研究评估了前沿 LLM(如 Claude Opus 5.5、GPT-6 Astra)与智能体框架(如 Claude Code、Codex、OpenCode)。实验显示最佳 LLM 智能体(Codex & GPT-6 Astra)总体成功率仅 21.05%,表明复杂验证任务仍有很大提升空间。
关键要点
- 01现有基准构造解耦子任务,偏离真实 UVM 验证工作流
- 02DVBench 含 95 个任务,覆盖 9 个功能域和 5 个难度等级
- 03每个任务要求智能体独立完成端到端验证流程
- 04评估最终测试平台编译、覆盖率与缺陷检测,并检查中间产物
- 05最佳智能体 Codex & GPT-6 Astra 总体成功率仅 21.05%
解读
尚无解读。
原始英文摘要
arXiv:2611.25066v1 Announce Type: new Abstract: Design verification (DV) remains a major bottleneck in hardware development, with large-scale verification using the Universal Verification Methodology (UVM) being highly labor-intensive. Recent advances in large language models (LLMs) and agentic systems offer new opportunities for DV automation. However, existing benchmarks primarily construct mutually decoupled subtasks, which deviates from the real-world workflow. In this paper, we present DVBench, the first end-to-end benchmark for Real-World UVM design verification agents. DVBench comprises 95 tasks across 9 functional domains and 5 difficulty levels, covering diverse verification scenarios and design scales. Each task requires agents to independently complete the full verification workflow. DVBench evaluates final testbenches via compilation, coverage, and bug detection, while assessing intermediate artifacts including feature lists, test plans, and reference models. We evaluate frontier LLMs (e.g. Claude Opus 5.5, GPT-6 Astra) and agent frameworks (e.g. Claude Code, Codex, OpenCode). Experimental results show that the best LLM-based agent (Codex \& GPT-6 Astra) can only achieve an overall success rate of 21.05\%, revealing substantial room for improvement in complex verification tasks.