何时停止惩罚动作？用于 VLA 强化学习的 Martingale GRPO
When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
Zongyao Zhao · Cheng-Jie Zhao · Ruihang Jiang · Bing Zheng · Zhaolin Wang · Yuanwei Liu
中文摘要
视觉-语言-动作(VLA)的一次失败轨迹并不意味着其中每个动作都对失败有贡献，但 GRPO 会把轨迹级优势分配给所有动作，可能连同用于纠错的恢复尝试也一并惩罚。通过成功概率鞅(policy-score martingale)刻画失败反馈的理想停止条件；在这些停止点，被移除梯度的条件期望为零，因此截断既可保持参考 GRPO 组梯度估计量的期望，也可能降低其协方差。当条件成功概率为零时即构成一种停止情形，可据此用错误与恢复线索选择反馈边界。Martingale GRPO(MART-GRPO)结合视觉语言模型(VLM)实现事件引导截断；该模型通过 LoRA 在标注训练集上微调，并在完整轨迹视频中定位最后一个可见错误且未见恢复的位置。失败反馈仅保留至所选错误区间，之后移除，同时维持成功轨迹损失项及原有损失归一化。实验分析不同回溯边界下的期望与方差；在 RoboTwin 2.0 上使用 OpenVLA-OFT 验证，MART-GRPO 在多项任务中的成功率均高于完整轨迹 GRPO。
关键要点
- 01指出 GRPO 将轨迹级优势施加于所有动作，会误罚有效的恢复尝试
- 02以成功概率鞅刻画理想停止条件，保证条件期望不变并可能降低协方差
- 03使用经 LoRA 微调的 VLM 定位错误及恢复事件，实现事件引导截断
- 04保留成功轨迹损失项与原始损失归一化，并分析回溯边界影响
- 05在 RoboTwin 2.0 多项任务上提升 OpenVLA-OFT 的成功率
解读
尚无解读。
原始英文摘要
arXiv:2610.32431v1 Announce Type: new Abstract: A failed vision-language-action (VLA) rollout does not imply that every action contributed to failure. Yet in group-relative policy optimization (GRPO), a single trajectory-level advantage is assigned to all actions in a rollout. Potentially useful recovery attempts can thus be penalized alongside the errors they aim to correct. Ideal stopping conditions for failure feedback are characterized using success-probability and policy-score martingales. At these stops, the removed gradient contribution has zero conditional expectation given the retained information. Truncation therefore preserves the expectation of a reference GRPO group-gradient estimator and can reduce its covariance. One such stop occurs when the conditional probability of success is zero, motivating error and recovery cues for selecting candidate feedback boundaries. Martingale GRPO (MART-GRPO) is introduced with a vision-language model (VLM) for event-guided truncation. The VLM is fine-tuned on our annotated training set using low-rank adaptation (LoRA). The last visible error without observed recovery is localized by the frozen VLM in complete rollout videos. Failure feedback is retained through the selected error interval and removed afterward, while successful-rollout loss terms and original loss normalization are preserved. For retrospective boundaries, changes in expectation and variance are analyzed to assess departures from ideal stopping. Experiments with OpenVLA-OFT on RoboTwin 2.0 show that MART-GRPO improves success rates over full-trajectory GRPO across multiple tasks.