跳到正文
返回论文列表
cs.CV提交于 已译

Keel:前沿模型通过重写权重来教导视觉语言模型

Keel: A Frontier Model Teaches a Vision-Language Model by Rewriting Its Weights

Jiajun Fan · Neil He · Bangji Yang · Qinglong Zheng · Ziwen Wang · Chaoran Cheng · et al.

中文摘要

前沿模型能否通过重写权重来修复视觉语言模型(VLM)?Qwen3.5 等小型推理 VLM 需要这样的修复:在思考模式下,它们会在许多图像问题上陷入循环,反复重复直到 token 上限,耗尽全部预算并丢失答案。常见补救措施作用于解码阶段,循环模型本身未变,或者必须被告知要改什么。然而原因写在权重里。Keel 是首个面向图像任务的 VLM 智能体式权重编辑方法,在权重中定位并移除该原因。仅被告知症状,作为智能体的前沿多模态 LLM 编辑器定位原因、编辑权重并测试,借助失败普查和仪器展示失败图像及 VLM 的注意力位置;在多个编辑器中,框架只保留能明显减少失败且不降低未见过问题准确率的编辑。修复无需训练,保持模型大小和每 token 成本,且适用于已编辑的权重。编辑器揭示了循环所在:从原始权重出发,两个模型上的独立编辑器都将其追溯到中间层的注意力输出,较小的 VLM 在循环内停止参考图像。在 Qwen3.5-0.8B 和 Qwen3.5-2B 上,跨六个图像基准(从视觉数学到 GUI grounding),编辑消除了每个基准上的大部分循环并缩短答案,编辑后的模型平均准确率显著高于思考模型;较大模型在 GUI grounding(练习问题中未出现的任务)上收益最大。由于修复存在于权重中,它与预算强制结合,在留出问题上更有利:在最佳可用权重上,Keel 在两个模型上均取得最佳平均准确率,在较大模型上显著高于我们比较的每个免训练基线。

关键要点

  1. 01问题:小型推理 VLM 在思考模式下对图像问题陷入循环,耗尽 token 预算并丢失答案,现有解码补救不触及权重根源。
  2. 02方法:Keel 用前沿多模态 LLM 作为智能体,仅凭症状定位并编辑权重,通过失败普查和注意力仪器筛选有效编辑。
  3. 03结果:在 Qwen3.5-0.8B/2B 六个图像基准上消除大部分循环并缩短答案,平均准确率显著高于思考模型。
  4. 04发现:独立编辑器将循环追溯到中间层注意力输出,较小 VLM 在循环内停止参考图像。
  5. 05局限:编辑与预算强制结合时仅在留出问题有利,且仅在两个模型和六个基准上验证。

解读

尚无解读。

原始英文摘要

arXiv:2611.25257v1 Announce Type: new Abstract: Can a frontier model repair a vision-language model (VLM) by rewriting its weights? Small reasoning VLMs such as Qwen3.5 need such a repair: in thinking mode they fall into loops on many image questions, repeating themselves until the token limit, spending the whole budget and losing the answer. The usual remedies act on the decoding, leaving the looping model as it was, or must be told what to change. Yet the cause is written in the weights. Keel, the first agentic weight-editing method for VLMs on image tasks, finds and removes it there. Told only the symptom, an editor, a frontier multimodal LLM acting as an agent, locates the cause, edits the weights and tests them, with a failure census and instruments showing it the failing images and where the VLM attends; among several editors, the harness keeps only edits that clearly reduce the failure without lowering accuracy on questions no editor saw. The repair needs no training, keeps the model's size and per-token cost, and applies to already-edited weights. The editors expose where the loops live: from the original weights, independent editors on both models trace them to the attention outputs of the middle layers, and the smaller VLM stops consulting the image inside a loop. On Qwen3.5-0.8B and Qwen3.5-2B, across six image benchmarks from visual mathematics to GUI grounding, the edit removes most loops on every benchmark and shortens answers, and the edited models are significantly more accurate than the thinking models on average; the larger one gains most on GUI grounding, a task absent from the practice questions. Because the repair lives in the weights, it combines with budget forcing where held-out questions favor it: on the best available weights, Keel has the best average accuracy on both models, significantly above every training-free baseline we compare on the larger one.

同方向论文 · cs.CV

查看全部 →