Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive inference cost due to large visual contexts and long decoding chains. We view this cost through two coupled axes: Visual Context Scaling (VCS), which controls how much visual evidence is passed to the language model, and Visual Reasoning Scaling (VRS), which controls how much inference-time reasoning search is performed. Existing methods typically optimize one axis at a time, leaving the joint allocation of compute across these axes underexplored. We introduce Adaptive Visual Inference Scaling (AVIS), a lightweight policy that adapts both VCS and VRS per query. AVIS realizes VCS through Key Diversity Visual (KDV) pruning, a training-free $O(N)$ key-based rule for removing redundant visual tokens before prefilling, and realizes VRS through adaptive self-consistency, using a learned difficulty predictor to select the number of reasoning rollouts. AVIS is deployment-friendly and compatible with shared-prefill inference, where all rollouts reuse a single prefilling pass and KV cache. Across diverse image and video reasoning benchmarks, AVIS improves the accuracy--compute trade-off relative to VCS-only and VRS-only baselines, and remains effective on top of RL post-trained VLMs while keeping compute and latency low.

翻译：现代视觉语言模型（VLM）从思维链提示和测试时缩放中获益，但这些优势往往伴随着因庞大的视觉上下文和冗长的解码链而带来的高昂推理成本。我们通过两个相互耦合的维度来审视这一成本：视觉上下文缩放（VCS）控制传递给语言模型的视觉证据量，而视觉推理缩放（VRS）则控制推理时执行的搜索量。现有方法通常每次只优化一个维度，忽视了跨这两个维度的计算资源联合分配。我们提出自适应视觉推理缩放（AVIS），这是一个轻量级策略，能够针对每个查询自适应地调整VCS和VRS。AVIS通过关键多样性视觉（KDV）剪枝实现VCS，这是一种无训练的O(N)基于键的规则，用于在预填充前移除冗余视觉标记；并通过自适应自一致性实现VRS，利用学习到的难度预测器选择推理展开次数。AVIS易于部署，兼容共享预填充推理，其中所有展开复用单一的预填充过程和KV缓存。在多种图像和视频推理基准测试中，相对于仅VCS和仅VRS的基线，AVIS改善了精度-计算权衡，并且在基于RL的后训练VLM上仍然有效，同时保持较低的计算和延迟。