跳到正文
返回论文列表
cs.RO提交于 已译

XBot-Omni:面向具身理解与交互的全模态模型

XBot-Omni: An Omni-Modal Model for Embodied Understanding and Interaction

Kerui Gu · Xing Fu · Shuoyang Sun · Cheng Peng · Jingru Duan · Huayu Gu · et al.

中文摘要

具身交互要求智能体将多模态感知与言语及非言语行为(包括语音、身体运动和面部表情)相结合,以有效传达意图。XBot-Omni 采用 Thinker–Talker–Actor 架构,将 Omni 范式扩展到机器人模态(身体运动和面部表情),其中“X”代表对机器人特定模态的可扩展性。Thinker 处理文本、图像、音频和视频以形成上下文适当的响应,然后由 Talker 和 Actor 分别通过语音和协调的身体表达来实现。在本报告中,创新性地引入“Actor”概念作为具身智能体的专用组件:一个协调的流式模块,在端到端流水线中利用 Thinker 和 Talker 的原生表示,仅引入适度的延迟开销。这种统一架构支持以全模态输入为条件的多模态生成,直接防止分阶段设计中的信息损失。在七个基准上全面评估 XBot-Omni,展示了在通用领域和以机器人中心的视频上强大的多模态理解能力,以及高质量、自然且协调的运动生成。此外,结果验证了其直接从全模态输入生成具身多模态响应的能力,满足具身交互的多样化需求。XBot-Omni 使机器人不仅能感知全模态世界,还能通过语音和身体行为表达自己,将 Omni 模型从对话交互扩展到物理环境中的具身交流。

关键要点

  1. 01问题:具身交互需整合多模态感知与言语/非言语行为,现有分阶段设计存在信息损失。
  2. 02方法:提出 Thinker–Talker–Actor 架构,将 Omni 范式扩展到机器人模态,Actor 为流式协调模块。
  3. 03结果:在七个基准上展示强多模态理解能力,并生成高质量、自然且协调的运动。
  4. 04结果:验证了直接从全模态输入生成具身多模态响应的能力,满足多样化交互需求。
  5. 05局限:Actor 模块引入适度延迟开销,但未具体量化。

解读

尚无解读。

原始英文摘要

arXiv:2611.17160v2 Announce Type: replace Abstract: Embodied interaction requires an agent to integrate multimodal perception with verbal and nonverbal behaviors to effectively communicate its intentions, including speech, body motion, and facial expressions. We present XBot-Omni, a Thinker--Talker--Actor architecture that extends the Omni paradigm to robotic modalities, including body motion and facial expressions, with ``X'' representing the extensibility to robotic-specific modalities. The Thinker processes text, images, audio, and video to formulate contextually appropriate responses, which are then realized by the Talker and Actor through speech and coordinated bodily expressions, respectively. In this report, we innovatively introduce the concept of ``Actor'' as a dedicated component for embodied agents: a coordinated, streaming module that leverages the native representations of the Thinker and Talker within an end-to-end pipeline, introducing only modest latency overhead. This unified architecture enables multimodal generation conditioned on omni-modal inputs directly to prevent information loss in staged designs. We comprehensively evaluate XBot-Omni across seven benchmarks, demonstrating strong multimodal understanding capabilities on both general-domain and robot-centric videos, as well as high-quality, natural, and well-coordinated motion generation. Furthermore, our results validate its capability to generate embodied multimodal responses directly from omni inputs, meeting the diverse requirements of embodied interaction. XBot-Omni enables robots not only to perceive the omni world but also to express themselves through speech and bodily behaviors, extending Omni models from conversational interaction to embodied communication in physical environments.

同方向论文 · cs.RO

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联工具 · 1

Google DeepMind76

Google DeepMind 发布 Gemini Robotics 2,为机器人带来全身智能控制

Google DeepMind 发布 Gemini Robotics 2,推出了三个模型:高级 VLA 模型可控制完整人形机器人从脚到指尖的运动,实现灵巧操作如系扣子;ER 2 推理模型可处理多步骤任务并协调多机器人协作;On-Device 2 模型可在几小时内适应全新机器人形态。

推荐理由:原文给出了三个模型的具体能力描述和可用入口,读者可以据此判断 Gemini Robotics 2 与前代及竞品的具体差异。