Vision-language models (VLMs) often generate massive visual tokens that greatly increase inference latency and memory footprint; while training-free token pruning offers a practical remedy, existing methods still struggle to balance local evidence and global context under aggressive compression. We propose Focus-Scan-Refine (FSR), a human-inspired, plug-and-play pruning framework that mimics how humans answer visual questions: focus on key evidence, then scan globally if needed, and refine the scanned context by aggregating relevant details. FSR first focuses on key evidence by combining visual importance with instruction relevance, avoiding the bias toward visually salient but query-irrelevant regions. It then scans for complementary context conditioned on the focused set, selecting tokens that are most different from the focused evidence. Finally, FSR refines the scanned context by aggregating nearby informative tokens into the scan anchors via similarity-based assignment and score-weighted merging, without increasing the token budget. Extensive experiments across multiple VLM backbones and vision-language benchmarks show that FSR consistently improves the accuracy-efficiency trade-off over existing state-of-the-art pruning methods. The source codes can be found at https://github.com/ILOT-code/FSR.

翻译：视觉语言模型（VLMs）通常生成大量视觉令牌，这显著增加了推理延迟和内存占用；尽管免训练令牌剪枝提供了一种实用的解决方案，但现有方法在激进压缩下仍难以平衡局部证据与全局上下文。我们提出聚焦-扫描-精炼（FSR），一种受人类启发的即插即用剪枝框架，它模拟人类回答视觉问题的方式：首先聚焦于关键证据，然后在需要时进行全局扫描，并通过聚合相关细节来精炼扫描到的上下文。FSR首先通过结合视觉重要性与指令相关性来聚焦关键证据，避免偏向视觉显著但与查询无关的区域。随后，它基于聚焦集条件扫描补充性上下文，选择与聚焦证据差异最大的令牌。最后，FSR通过基于相似度的分配和分数加权合并，将附近的信息性令牌聚合到扫描锚点上，从而精炼扫描上下文，且不增加令牌预算。在多个VLM主干和视觉语言基准上的广泛实验表明，FSR在准确性与效率的权衡上持续优于现有的最先进剪枝方法。源代码可在 https://github.com/ILOT-code/FSR 找到。