自回归视觉语言模型中视觉对象定位的角度监督
Angular Supervision for Visual Object Grounding in Autoregressive Vision-Language Models
Parsa Esmaeilkhani · Longin Jan Latecki
中文摘要
自回归视觉语言模型(VLMs)难以将生成文本对应到正确的图像区域；现有可解释性方法如注意力图和 Logit Lens 又常因文本到视觉的注意力较弱及 sink token 存在，产生较差的空域解释。提出角度相关性(Angular Relevance，$R_L$)，以视觉 token 嵌入与反嵌入矩阵行之间的余弦相似度作为简单可视化指标。由于该指标不受 token 范数影响，在无需训练的现成模型上，可生成比注意力图或 Logit Lens 更锐利、更忠实的热力图，并在指代表达分割(RES)任务上取得有竞争力的结果，优于既有诊断图和基于 CLIP 的免训练方法。为在跨模态混合中保留局部对象信息，提出角度对象感知损失(Angular Object-Aware Loss，AOL)，直接监督视觉 token 嵌入，使其与对应对象概念的词表方向在角度上对齐。结合标准下一 token 预测目标进行微调后，热力图中的对象定位显著变清晰；无需训练分割头，也不会降低通用视觉问答(VQA)性能，所学定位还能迁移到训练中未出现的对象类别级分割。
关键要点
- 01现有注意力图与 Logit Lens 空间解释较差，受弱文本—视觉注意力和 sink token 影响
- 02角度相关性以余弦相似度衡量视觉 token 与反嵌入矩阵行的关联
- 03免训练的角度相关性可生成更锐利热力图，并优于既有诊断图及 CLIP 方法
- 04AOL 强化视觉 token 与对象概念词表方向的角度对齐，改善对象定位
- 05定位能力可迁移至未见对象类别，且不损害通用 VQA 性能
解读
尚无解读。
原始英文摘要
arXiv:2610.32220v1 Announce Type: new Abstract: Autoregressive vision-language models (VLMs) struggle to ground generated text in the correct image regions, and existing explainability methods, such as attention maps and Logit Lens, often produce poor spatial explanations due to weak text-to-vision attention and sink tokens. We propose \textbf{Angular Relevance} ($R_L$), a simple visualization measure based on cosine similarity between visual token embeddings and unembedding-matrix rows. Because $R_L$ is invariant to token norm, it produces substantially sharper and more faithful heatmaps than attention maps or Logit Lens on out-of-the-box models. Without any training, $R_L$ yields competitive referring-expression segmentation (RES), outperforming prior diagnostic maps and CLIP-based training-free methods. To preserve localized object information during cross-modal mixing, we introduce \textbf{Angular Object-Aware Loss (AOL)}, an auxiliary loss that directly supervises visual token embeddings to align angularly with the vocabulary directions of their corresponding object concepts. Finetuning with AOL alongside the standard next-token prediction objective significantly sharpens object localization in heatmaps, showing that the VLM's own visual tokens can be used directly for grounding and improving RES, without training any segmentation heads and without degrading general VQA performance. We further demonstrate that the learned grounding also transfers to category-level segmentation of object categories not seen during training.