ScaleDiag:诊断 GUI 定位中测试时扩展的极限
ScaleDiag: Diagnosing the Limits of Test-Time Scaling in GUI Grounding
Chen Liu · Zhen Fang · Xin Yu · Dadong Wang · Ling Chen
中文摘要
GUI 定位(GUI grounding)将截图与自然语言指令映射到目标界面元素的坐标。现有方法通过视觉扩展、指令扩展和采样扩展来提升定位效果,但这些策略究竟解决了哪些定位瓶颈、其收益在何处饱和仍不清楚。为此,ScaleDiag 作为一个受控诊断框架,分别独立改变目标尺度与周围上下文、目标特定信息与辅助描述、候选覆盖与选择,从而隔离视觉输入、指令信息和预测采样对 GUI 定位的影响。在五个基准和最多八个视觉语言模型上评估发现,每种扩展策略的有效区间都远窄于通常假设。(1) 视觉扩展受目标占据比例支配而非固定裁剪比例,当目标仅占裁剪视图的 0.32% 时全屏错误通常即可被挽救,继续放大会变得有害。(2) 指令扩展由识别目标的信息驱动而非描述丰富度,即使来自真值的颜色、大小、形状等属性也常为中性或有害,而外观与空间描述贡献了超过 90% 的可达增益。(3) 重复采样暴露了不同局限:正确坐标常被生成但难以选中,留下 5.1–11.0 分的 oracle 余量,而几何共识几乎无改善。这些结果表明测试时扩展受限于不同的运行区间,而非从更多视觉聚焦、更丰富指令或更多样本中单调获益。代码见 https://github.com/YenanLiu/ScaleDiag。
关键要点
- 01视觉扩展由目标占据比例而非固定裁剪比例决定,目标占 0.32% 时即可挽救全屏错误,继续放大反而有害。
- 02指令扩展的增益主要来自外观与空间描述(超 90%),颜色、大小、形状等真值属性常为中性或有害。
- 03重复采样中正确坐标常被生成但难以选中,留下 5.1–11.0 分 oracle 余量,几何共识几乎无改善。
- 04在五个基准和最多八个视觉语言模型上验证,各扩展策略的有效区间远窄于通常假设。
解读
尚无解读。
原始英文摘要
arXiv:2611.25297v1 Announce Type: new Abstract: GUI grounding maps a screenshot and a natural language instruction to the coordinates of the intended interface element. Existing approaches improve grounding through visual scaling, instruction scaling, and sampling scaling, yet it remains unclear which grounding bottlenecks these strategies address and where their gains saturate. To address this gap, we introduce ScaleDiag, a controlled diagnostic framework that isolates the effects of visual input, instruction information, and prediction sampling on GUI grounding by independently varying target scale and surrounding context, target-specific information and auxiliary descriptions, and candidate coverage and selection, respectively. We evaluate ScaleDiag across five benchmarks and up to eight vision-language models and find that each scaling strategy operates within a much narrower regime than commonly assumed. (1) Visual scaling is governed by target occupancy rather than a fixed crop ratio, with full-screen errors typically rescued once the target occupies only 0.32% of the cropped view, while further zooming eventually becomes harmful. (2) Instruction scaling is driven by information that identifies the target rather than description richness itself, as even ground-truth-derived attributes such as color, size, and shape are often neutral or harmful, while appearance and spatial descriptions account for over 90% of the achievable gain. (3) Repeated sampling exposes a different limitation, as correct coordinates are frequently generated but remain difficult to select, leaving 5.1--11.0 points of oracle headroom while geometric consensus provides almost no improvement. Together, these results show that test-time scaling is constrained by distinct operating regimes rather than benefiting monotonically from more visual focus, richer instructions, or additional samples. Our code is available at https://github.com/YenanLiu/ScaleDiag.