先改写再行动:视觉-语言-动作模型的语言敏感性表征与缓解
Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models
Mikey Watts (Independent Researcher) · Yuchen Cui (University of California · Los Angeles)
中文摘要
视觉-语言-动作模型(VLA)对指令措辞极为敏感,且并未继承其底层视觉-语言模型的语言鲁棒性。仅仅改动一个词就可能导致成功率大幅波动:π₀.₅ 在 LIBERO 炉灶任务上对"switch on the stove"达到 100% 成功率,而对"switch on the hot plate"仅 2%;即便用改写增广微调的 π₀ 检查点,波动幅度仍可达 61 个百分点。通过具有统计检验的单次编辑波动与 oracle 短语搜索对此敏感性进行表征,结果表明措辞本身几乎弥合了分布内与分布外任务之间 21 个百分点的差距。随后在不修改策略的前提下缓解该问题。鉴于这种敏感性具有系统性,可被表达为显式规则:对少量训练任务的多种措辞打分,由大语言模型将证据蒸馏为十条到二十条改写规则,部署时按这些规则对每条输入指令改写一次。该方法在十二个保留任务上(涵盖对抗性、VLM 生成与人类生成的措辞)将冻结 π₀ 相对提升 16% 到 27%,增益集中在分布外任务。在 π₀.₅ 与 LIBERO 上复现该流水线,使微调内成功率从 93.6% 提升至 97.8%。该方法无需重训练,无需逐步验证,并可零样本迁移至未见过的任务和指令。项目页面:https://sttawm.github.io/rephrase-before-you-act
关键要点
- 01问题:VLA 对指令措辞极度敏感,单词改动可造成最高 61 个百分点成功率波动,措辞差异本身几乎弥合 21 个百分点的分布内外任务差距
- 02方法:不改策略,在大规模措辞打分基础上由 LLM 蒸馏十条到二十条改写规则,部署时对每条指令按规则改写一次,无需重训练或逐步验证
- 03结果:在对抗性、VLM 生成、人类生成措辞的十二个保留任务上将冻结 π₀ 相对提升 16%–27%;π₀.₅ 在 LIBERO 上微调内成功率从 93.6% 提升至 97.8%
- 04特性:系统可零样本迁移至未见过的任务与指令,增益集中在分布外任务
解读
尚无解读。
原始英文摘要
arXiv:2610.10526v1 Announce Type: new Abstract: Vision-language-action models (VLAs) are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-language models they are built on. A one-word edit can move success by tens of points: $\pi_{0.5}$ turns on a LIBERO stove 100% of the time for "switch on the stove" and 2% for "switch on the hot plate", and a $\pi_0$ checkpoint finetuned with rephrase augmentation still shows swings of up to 61 points. We characterize this sensitivity with statistically tested single-edit swings and an oracle phrase search, which shows that phrasing alone nearly closes the 21-point gap between in-distribution and out-of-distribution tasks. We then reduce it without modifying the policy. Because the sensitivity is systematic, it can be expressed as explicit rules: we score many phrasings of a few training tasks, have a large language model distill the evidence into ten to twenty rephrasing rules, and at deployment rewrite each incoming instruction once under these rules. The rules improve the frozen $\pi_0$ by 16 to 27% relative on twelve held-out tasks across adversarial, VLM-generated, and human-generated phrasings, with gains concentrated on out-of-distribution tasks. The pipeline replicates on $\pi_{0.5}$ and LIBERO, lifting in-finetune success from 93.6% to 97.8%. The method requires no retraining and no per-step verification, and applies zero-shot to unseen tasks and instructions. Project website: https://sttawm.github.io/rephrase-before-you-act