WorldFlow：通过大规模预训练联合预测视频与3D运动
WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
Yuhao Lu · Jiongxu Chen · Hexiang Wei · Jiajin Wu · Juelin Li · Mingjian Liang · et al.
中文摘要
预测未来物理状态是具身智能的核心挑战。现有视频世界模型将3D动态隐式处理，基于动作的公式与特定机器人配置绑定，3D运动预测模型仅使用流监督，为物理动态提供有限学习信号。为此，提出WorldFlow，一种联合预测未来视频和基于查询的3D轨迹的生成式世界模型。将3D运动参数化为与视频网格对齐的连续位移token，统一Mixture-of-Transformers（MoT）主干在rectified-flow目标下联合去噪两种模态。该公式将显式几何动态与视觉演化耦合，有效释放预训练视频模型中嵌入的物理常识。为实证扩展和验证该统一架构，在包含模拟和真实世界领域的144万对象级样本异构混合上训练WorldFlow。进一步引入WorldFlowBench，包含来自11个数据源的1,678个评估样本，涵盖模拟和真实世界交互，包括空间推理任务。在所有三个评估集上，WorldFlow在评估的基于学习方法中取得最低平均位移误差（ADE）和最终位移误差（FDE）。在具有挑战性的SpatialCog集上，WorldFlow的ADE为0.060 m，与最强外部基线相比降低54.2%。与仅流变体相比，ADE和FDE分别降低55.9%和68.5%。项目页面：https://worldflow-rbs.github.io/。
关键要点
- 01现有视频世界模型隐式处理3D动态，动作公式绑定特定机器人，3D运动预测仅用流监督信号有限。
- 02将3D运动参数化为与视频网格对齐的连续位移token，用统一MoT主干在rectified-flow目标下联合去噪视频与轨迹。
- 03在144万对象级样本的异构混合上训练，涵盖模拟与真实世界领域。
- 04提出WorldFlowBench，含11个数据源1,678个评估样本，覆盖空间推理任务。
- 05在SpatialCog集上ADE为0.060 m，较最强外部基线降低54.2%，较仅流变体ADE和FDE分别降低55.9%和68.5%。
解读
尚无解读。
原始英文摘要
arXiv:2610.39278v1 Announce Type: cross Abstract: Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2\% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9\% and 68.5\%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.