跳到正文
返回论文列表
cs.CL提交于 已译

跨智能体交接的任务接管基准测试

Benchmarking Task Takeover Across Agent Handoffs

Wenyuan Li · Guang Li · Dongyuan Li · Keisuke Maeda · Takahiro Ogawa · Miki Haseyama

中文摘要

在智能体协作长任务时,责任可能在任务完成前在模型间转移。接收模型需基于先前的决策、观察和行动继续工作。现有评估仅关注任务成功率或记忆保留,未能捕捉交接信息保留与后续工作延续之间的关系。HSC-Bench 包含 1,039 个任务,评估任务接管的两个互补方面:恢复相关代码库证据,以及从既定的对话、工具和文件工作状态继续。对六个接收模型的比较显示,结构化交接状态提升了所有模型的代码库证据恢复能力,并提升了大多数模型的延续性能。两个赛道(证据恢复与状态延续)的最强接收模型不同,表明仅凭证据恢复排名无法表征从继承状态继续时的性能。HSC-Bench 结合后续响应和行动检查,评估了对先前证据的访问以及对既定承诺和已完成工作的遵守,为根据需延续的工作选择接收模型和评估交接表示提供了基础。

关键要点

  1. 01现有评估仅关注任务成功率或记忆保留,未能捕捉交接信息保留与后续工作延续之间的关系。
  2. 02HSC-Bench 包含 1,039 个任务,评估恢复代码库证据和从既定对话、工具、文件状态继续两个互补方面。
  3. 03结构化交接状态提升了所有接收模型的代码库证据恢复能力,并提升了大多数模型的延续性能。
  4. 04证据恢复与状态延续两个赛道的最强接收模型不同,表明仅凭证据恢复排名无法表征从继承状态继续时的性能。
  5. 05HSC-Bench 结合后续响应和行动检查,评估了对先前证据的访问以及对既定承诺和已完成工作的遵守。

解读

尚无解读。

原始英文摘要

arXiv:2610.36523v1 Announce Type: new Abstract: As agents collaborate on longer tasks, responsibility may pass between models before the work is complete. The receiving model must build on prior decisions, observations, and actions while addressing the remaining requirements. Evaluating this transition requires connecting the information preserved at handoff to how the receiver continues the work, a relationship that task-success and memory-retention scores alone do not capture. We introduce the Handoff State Continuity Benchmark (HSC-Bench), comprising 1,039 tasks that evaluate two complementary aspects of task takeover: recovering relevant repository evidence and continuing from established conversational, tool, and file-work state. Comparisons across six receiving models show that structured handoff state improves repository evidence recovery for every model and continuation performance for most models. The strongest receivers differ between the two tracks, showing that evidence-recovery rankings alone do not characterize performance when continuing from inherited state. By combining these comparisons with checks of subsequent responses and actions, HSC-Bench evaluates both access to prior evidence and adherence to established commitments and completed work. This provides a basis for selecting receiving models and assessing handoff representations according to the work that must be carried forward.

同方向论文 · cs.CL

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联提示词 · 2

写作github.com/f/awesome-chatgpt-prompts·外部收录·

project marketing

Act as a Notion Content Automation Expert. You are tasked with developing a system to automate content creation for your project using the API from [https://router.bynara.id/dashboard](https://router.bynara.id/dashboard). You will utilize 5…

学习github.com/f/awesome-chatgpt-prompts·外部收录·

Spec Interview

read this${specmd:spec.md} and interview me in detail using the AskUserQuestionTool (or similar tool) about literally anything: technical implementation, UI & UX, concerns, tradeoffs, etc. but make sure the questions are not obvious be ver…