跳到正文
返回论文列表
cs.AI提交于 已译

Long-Horizon-Terminal-Bench 2.0:自动化多样化任务构建与衡量智能体持续进展

Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

Zongxia Li · Zhongzhi Li · Yucheng Shi · Ruhan Wang · Junyao Yang · Chengsong Huang · et al.

中文摘要

长时程终端基准旨在衡量智能体能否通过过程奖励在扩展工作流中维持进展,而非仅检查最终状态。Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0) 保留了46个原始任务,修复了缺陷并扩展了31个任务,共包含13个领域的77个任务。扩展任务通过系统性重构管道生成:从可用工件出发,利用参数化参考实现生成场景,展示6个作为行为证据,保留3个用于评估,并使用密集进度阶梯奖励部分进展以防止记忆。评估时长从90分钟延长至3小时,每30分钟记录验证点。在15个前沿模型的1,145次运行中,GPT-6-astra以0.69的密集平均奖励领先,37个任务得分≥0.95;GPT-5.6-sol紧随其后。在30个重构任务中,二元验证器平均分0.221,而密集阶梯平均分0.653,表明仅结果评估丢弃了大量中间进展。LHTB 2.0提供了更广泛、强化且过程感知的测试平台。

关键要点

  1. 01LHTB 2.0包含13个领域的77个任务,修复了原有缺陷并扩展了31个任务。
  2. 02采用系统性重构管道生成任务,利用密集进度阶梯奖励部分进展以防止记忆。
  3. 03评估时长延长至3小时,每30分钟记录验证点以直接测量持续进展。
  4. 04GPT-6-astra在15个前沿模型中表现最佳,密集平均奖励为0.69。
  5. 05密集阶梯评分(0.653)显著高于二元验证器评分(0.221),凸显中间进展价值。

解读

尚无解读。

原始英文摘要

arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its final state passes a binary check. We present Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0), an extension of Long-Horizon-Terminal-Bench that preserves its 46 original tasks while correcting task and evaluation-infrastructure defects and extending 31 tasks. LHTB 2.0 contains 77 tasks across 13 domains and eight task types. The extended tasks are produced through a systematic reconstruction-task pipeline. We start from a working scientific or engineering artifact, then use a parameterized reference implementation to generate multiple scenarios, reveal six as behavioral evidence, hold out three for evaluation, and use dense progress ladders to reward partial progress while preventing memorization of public outputs. We also extend the evaluation horizon from 90 minutes to three hours and record verifier checkpoints every 30 minutes, enabling direct measurement of sustained progress within a run. Across 1,145 scored model-task runs from 15 frontier models, GPT-6-astra leads with dense mean reward 0.69 and 37 tasks at R at least 0.95, followed by GPT-5.6-sol with dense mean reward 0.67 and 32 tasks at the same threshold. On the 30 reconstruction tasks, binary verifier scores average 0.221, whereas dense ladder scores average 0.653, showing that outcome-only evaluation discards substantial intermediate progress. LHTB 2.0 provides a broader, hardened, and process-aware testbed for studying reliable agent execution over hours.

同方向论文 · cs.AI

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联工具 · 1

Hugging Face Blog71

Microsoft 与 Hugging Face 用数据库终态奖励后训练 ThinkingBox 智能体

Microsoft 与 Hugging Face 将 ThinkingBox 的数据库终态检查转化为训练奖励,并对 Qwen3.8-27B 进行 50 步 GRPO 微调。

推荐理由:文章把数据库终态检查转化为训练奖励,公开完整训练流程,并同时量化重复成功率、覆盖率与成本变化。

关联提示词 · 1