EgoGeneralist：通过视频原生建模统一第一人称动作-视频任务
EgoGeneralist: Unifying Egocentric Action-Video Tasks through Video-Native Modeling
Yanlin Jin · Bowen Yang · Cheng Zhang · Yifan Yang · Zhe Li · Jianfei Cai · et al.
中文摘要
现代视频模型具备强视觉先验，但无法在度量3D空间中原生表示人类动作。EgoGeneralist将预训练视频生成器适配为统一动作条件的第一人称视频生成、姿态估计与联合动作-视频生成。角色自适应动作渲染器为多向动作-视频学习提供视频原生表示，使度量运动既可作为空间控制信号，也可作为可恢复的生成目标。跨流动作-视频耦合通过轻量适配在度量运动与视觉动态之间建立紧密对应，同时保留预训练视觉先验。这些设计共同构成覆盖五类动作-视频任务的通用框架。尽管仅在受控实验室录制数据上适配，EgoGeneralist仍能泛化到多样未见场景与交互，具备有效的动作控制与视觉质量，弥合了预训练视觉知识与度量3D人体运动之间的差距。
关键要点
- 01问题：现有视频模型缺乏对度量3D空间人体动作的原生表示能力
- 02方法：用角色自适应动作渲染器提供视频原生表示，统一五类动作-视频任务
- 03方法：跨流动作-视频耦合以轻量适配对齐度量运动与视觉动态，保留预训练先验
- 04结果：仅在实验室数据适配，仍泛化到未见场景与交互，动作控制与视觉质量有效
解读
尚无解读。
原始英文摘要
arXiv:2611.08477v1 Announce Type: new Abstract: Modern video models possess strong visual priors, yet do not natively represent human action in metric 3D space. We present EgoGeneralist, which adapts pretrained video generators to unify action-conditioned egocentric video generation, pose estimation, and joint action-video generation. Our role-adaptive action renderer provides video-native representations for multidirectional action-video learning, making metric motion usable as both a spatial control signal and a recoverable generation target. Cross-stream action-video coupling establishes close correspondence between metric motion and visual dynamics through lightweight adaptation while preserving the pretrained visual prior. Together, these designs establish a generalist framework spanning five action-video tasks. Despite adaptation only on controlled laboratory recordings, EgoGeneralist generalizes to diverse unseen scenes and interactions with effective action control and visual quality, bridging pretrained visual knowledge and metric 3D human motion.