跳到正文
返回论文列表
cs.RO提交于 已译

DexSeed:将人手-物体交互扩展为机器人演示以学习灵巧策略

DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning

Huaijin Pi · Zehong Shen · Yong Wang · Kechun Xu · Changpeng Wang · Dingkun Wei · et al.

中文摘要

研究如何为灵巧视觉-语言-动作(VLA)学习生成多样且物理可执行的机器人演示。现有演示生成方法通常依赖昂贵的遥操作示例;人手-物体数据集虽可复用,但现有迁移方法主要复现给定交互。将这些交互转为 VLA 训练数据还需覆盖不同物体与操作位置,并具备可执行的臂与手指动作及视觉观测。为此提出 DexSeed,一个从人手-物体交互低成本生成灵巧机器人数据的框架。核心思路是在物理仿真中扩展人类序列,同时保留抓取方式并在臂跟踪动力学下实现所得运动。具体按抓取模式组织源序列并共享语言描述;基于 InteractionMesh 的重定向与臂轨迹增强使序列适配机器人手、不同物体几何与拾放位置,同时保留局部手-物体关系。RL 手指控制策略产生可执行 rollout,臂跟踪预计算轨迹。成功 rollout 的渲染观测与记录动作同抓取描述配对,用于微调 VLA。实验显示重定向接触保真度提升,整臂执行比评估基线更可靠;仿真评估与真实部署进一步表明,仅用生成演示微调的 VLA 能执行语言指定的抓取。

关键要点

  1. 01问题:灵巧 VLA 演示依赖昂贵遥操作,人手数据迁移方法仅复现给定交互,缺乏物体与位置覆盖。
  2. 02方法:按抓取模式组织人手序列,用 InteractionMesh 重定向与臂轨迹增强适配机器人手和物体几何。
  3. 03方法:RL 手指策略生成可执行 rollout,臂跟踪预计算轨迹,渲染观测与动作配对微调 VLA。
  4. 04结果:重定向接触保真度提升,整臂执行比基线更可靠,仅用生成数据微调的 VLA 可执行语言指定抓取。

解读

尚无解读。

原始英文摘要

arXiv:2611.14092v1 Announce Type: new Abstract: We study how to generate diverse, physically executable robot demonstrations for dexterous vision-language-action (VLA) learning. Existing demonstration-generation methods typically rely on costly teleoperated examples. Meanwhile, human hand--object datasets provide a reusable source, but existing transfer methods mainly reproduce given interactions. Turning these interactions into VLA training data further requires coverage of different objects and manipulation locations, together with executable arm and finger actions, and visual observations. To achieve this, we present DexSeed, a framework for low-cost generation of dexterous robot data from human hand--object interactions. Our key idea is to expand human sequences while preserving their grasps and realizing the resulting motions under arm-tracking dynamics in physics simulation. Specifically, we organize source sequences by grasp mode with shared language descriptions. InteractionMesh-based retargeting and arm trajectory augmentation adapt these sequences to the robot hand, different object geometries, and pick-and-place locations while preserving local hand--object relationships. An RL finger-control policy produces executable rollouts, while the arm tracks precomputed trajectories. Rendered observations and recorded actions from successful rollouts are paired with grasp descriptions to fine-tune a VLA. Experiments show improved contact fidelity in retargeting and more reliable whole-arm execution than the evaluated baselines. Simulation evaluation and real-world deployment further demonstrate that a VLA fine-tuned solely on the generated demonstrations can execute language-specified grasps. Project page: https://robbyant-research.github.io/DexSeed/.

同方向论文 · cs.RO

查看全部 →