Despite rapid progress, pretrained vision-language models still struggle when answers depend on tiny visual details or on combining clues spread across multiple regions, as in documents and compositional queries. We address this by framing grounding as test-time evidence retrieval: given a query, the model should actively identify where to look next to resolve ambiguity. To this end, we propose a training-free, model-intrinsic grounding method that uses uncertainty as supervision. Specifically, we compute the entropy of the model's next-token distribution and backpropagate it to the visual token embeddings to obtain an entropy-gradient relevance map, without auxiliary detectors or attention-map heuristics. We then extract and rank multiple coherent regions to support multi-evidence queries, and introduce an iterative zoom-and-reground procedure with a spatial-entropy stopping rule to avoid over-refinement. Experiments on seven benchmarks across four VLM architectures demonstrate consistent improvements over existing methods, with the largest gains on detail-critical and high-resolution settings, while also producing more interpretable evidence localizations.

翻译：尽管预训练视觉语言模型取得了快速进展，但当答案依赖于微小视觉细节或需要综合多个区域散布线索时（如文档和组合性查询），其表现仍有不足。我们将视觉定位重新定义为测试时证据检索任务：面对查询时，模型应主动识别下一步需关注的区域以消除歧义。为此，我们提出了一种无需训练、基于模型内在特性的定位方法，以不确定性作为监督信号。具体而言，我们计算模型下一词元分布的熵，并通过反向传播将其作用于视觉词元嵌入，从而获得熵梯度相关性热图，无需辅助检测器或注意力图启发式方法。随后，我们提取并排序多个连贯区域以支持多证据查询，并引入具有空间熵停止准则的迭代缩放与重新定位过程，避免过度细化。在四种视觉-语言模型架构的七个基准测试上的实验表明，本方法持续优于现有方法，在细节关键型和高分辨率场景中提升最为显著，同时生成的证据定位更具可解释性。