NVIDIA 在 CoRL 发布 GR00T 2 世界动作模型与 Isaac Lab 3.0
NVIDIA 在 CoRL 上发布基于 Cosmos 构建的开放世界动作模型 GR00T 2,训练数据包含 10 万小时开源人类第一视角数据,在 50 项任务中仅用单次演示即达到 71% 平均成功率,五次演示提升至 95%。
推荐理由:NVIDIA 在 CoRL 集中放出 GR00T 2、Isaac Lab 3.0 与多个基准,读者可据此了解机器人仿真训练工具链的最新形态。
What Matters for World Modeling in Vision-Language-Action Models?
Qisen Ma · Yixiang Chen · Yuan Xu · Peiyan Li · Jiabing Yang · Kai Wang · et al.
将世界模型集成到视觉-语言-动作模型(VLA)中,使策略能预测未来动态,超越了反应式的观察-动作映射。然而,现有方法在世界表征、集成范式和策略骨干上差异巨大,难以确定有效集成的关键。研究将现有工作分为四种范式:隐式未来对齐、显式未来查询、MoT未来专家及外部世界预测器。在统一框架下,结合多种世界表征与策略骨干,在域内、域外、长时程、动态、少样本及真实场景中系统评估。主要发现:1) 世界表征有效性依赖范式,2D语义表征提供跨范式的稳健增益;2) 无单一最优范式,解耦的MoT未来专家整体表现具竞争力,尤其在域外鲁棒性上;3) 世界建模始终有益于基于标准VLM的策略,解耦设计与预训练VLA兼容性更佳;4) 相比仅动作基线,世界建模在多场景提升性能,长时程与动态任务增益显著。
尚无解读。
arXiv:2610.19825v1 Announce Type: new Abstract: Integrating world models into Vision-Language-Action models (VLAs) has emerged as a promising direction, enabling policies to anticipate future dynamics beyond reactive observation-to-action mapping. However, existing approaches vary simultaneously in their world representations, integration paradigms, and policy backbones, making it difficult to determine what truly matters for effective world model integration. To disentangle these factors, we categorize existing efforts into four representative paradigms, namely Implicit Future Alignment, Explicit Future Query, MoT Future Expert, and External World Predictor. We instantiate each within a unified framework and systematically evaluate these paradigms with diverse world representations and policy backbones across in-domain, out-of-domain, long-horizon, dynamic, few-shot, and real-world settings. Our study yields four main findings: 1) world representation effectiveness is paradigm-dependent, whereas 2D semantic representations offer robust, cross-paradigm gains; 2) no single paradigm is universally optimal, yet the decoupled MoT Future Expert offers competitive overall performance, particularly in OOD robustness; 3) world modeling consistently benefits policies built on standard VLMs, while decoupled designs exhibit greater compatibility with pretrained VLAs; and 4) world modeling improves performance over action-only baselines across multiple settings, with notable gains on long-horizon and dynamic tasks. Project page and video demonstrations are available at https://dylanmqs.github.io/WM-in-VLAs/.
NVIDIA 在 CoRL 上发布基于 Cosmos 构建的开放世界动作模型 GR00T 2,训练数据包含 10 万小时开源人类第一视角数据,在 50 项任务中仅用单次演示即达到 71% 平均成功率,五次演示提升至 95%。
推荐理由:NVIDIA 在 CoRL 集中放出 GR00T 2、Isaac Lab 3.0 与多个基准,读者可据此了解机器人仿真训练工具链的最新形态。