混合计算机使用：面向通用智能体的统一程序化、语义与视觉接口
Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents
Haoqing Wang · Xiaoyi Zhang · Sicheng Fan · Zhexin Zhang · Xinjie Zhang · Jiahao Li · et al.
中文摘要
计算机使用智能体（computer-use agent）正承担从日常桌面操作到长周期专业工作流的任务。然而，以截图和鼠标键盘操作为中心的接口会引入人类交互先验，要求反复解读屏幕，同时使计算机的语义与程序化能力未被充分暴露。主张暴露这些能力并让模型自行选择与组合。提出 Hybrid Computer Use（HCU），一种面向通用智能体的统一接口，融合三条互补通道：通过代码与命令行工具的程序化交互、通过无障碍（accessibility, AX）的语义交互、以及通过图形用户界面（GUI）的视觉交互。HCU 支持高效的 AX 交互，并在通道切换时防止过期状态动作。配套的 HCU skill 指导智能体选择与协调通道以有效使用统一接口。所设计的 AX 工具返回紧凑观测、批量执行动作并以 AX diff 报告效果，从而减少模型往返次数。在 OSWorld 2.0 上，HCU 在对比系统中取得最高的 Partial 与 Binary 分数。分析智能体轨迹发现，所有被评估模型均因 HCU 而提升，但各模型在通道间的工作分配不同，且在效率与任务成功率上均持续优于对比基线，凸显允许灵活选择与组合通道的统一计算机使用接口的价值。
关键要点
- 01截图加鼠标键盘的接口强加人类交互先验，反复解读屏幕且未暴露语义与程序化能力
- 02HCU 统一程序化、AX 语义与 GUI 视觉三条通道，并防止切换时的过期状态动作
- 03AX 工具返回紧凑观测、批量执行并以 AX diff 报告效果，减少模型往返次数
- 04在 OSWorld 2.0 上取得对比系统中最高的 Partial 与 Binary 分数
- 05各模型均因 HCU 提升，但通道工作分配不同，效率与成功率均优于基线
解读
尚无解读。
原始英文摘要
arXiv:2610.41432v1 Announce Type: cross Abstract: Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed. We argue for exposing these capabilities and letting models choose and combine them. We introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels: programmatic interaction through code and command-line tools, semantic interaction through accessibility (AX), and visual interaction through graphical user interfaces (GUI). Our HCU supports efficient AX interaction and guards against stale-state actions during channel switching. A supplementary HCU skill guides agents in selecting and coordinating channels to use the unified interface effectively. Our designed AX tools return compact observations, execute actions in batches, and report their effects as AX diffs, so the model needs fewer round trips. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. Analyzing agent trajectories, we find that all evaluated models improve with HCU yet allocate work across the channels differently and consistently outperform compared baselines in terms of both efficiency and task success, highlighting the value of unified computer-use interfaces that allow flexible channel selection and combination.