In recent years, large-scale vision-language models (VLMs) have demonstrated remarkable performance on multimodal understanding and reasoning tasks. However, handling high-dimensional visual features often incurs substantial computational and memory costs. VoCo-LLaMA alleviates this issue by compressing visual patch tokens into a few VoCo tokens, reducing computational overhead while preserving strong cross-modal alignment. Nevertheless, such approaches typically adopt a fixed compression rate, limiting their ability to adapt to varying levels of visual complexity. To address this limitation, we propose Adaptive-VoCo, a framework that augments VoCo-LLaMA with a lightweight predictor for adaptive compression. This predictor dynamically selects an optimal compression rate by quantifying an image's visual complexity using statistical cues from the vision encoder, such as patch token entropy and attention map variance. Furthermore, we introduce a joint loss function that integrates rate regularization with complexity alignment. This enables the model to balance inference efficiency with representational capacity, particularly in challenging scenarios. Experimental results show that our method consistently outperforms fixed-rate baselines across multiple multimodal tasks, highlighting the potential of adaptive visual compression for creating more efficient and robust VLMs.

翻译：近年来，大规模视觉语言模型在多模态理解与推理任务中展现出卓越性能。然而，处理高维视觉特征通常会产生巨大的计算与内存开销。VoCo-LLaMA通过将视觉补丁令牌压缩为少量VoCo令牌来缓解这一问题，在保持强大跨模态对齐能力的同时降低了计算负担。然而，此类方法通常采用固定的压缩率，限制了其适应不同视觉复杂度水平的能力。为解决这一局限，我们提出自适应VoCo框架，通过轻量级预测器增强VoCo-LLaMA以实现自适应压缩。该预测器利用视觉编码器中的统计线索（如补丁令牌熵和注意力图方差）量化图像的视觉复杂度，从而动态选择最优压缩率。此外，我们提出融合速率正则化与复杂度对齐的联合损失函数，使模型能够在推理效率与表征能力之间取得平衡，特别是在复杂场景中。实验结果表明，我们的方法在多个多模态任务中持续优于固定压缩率基线，凸显了自适应视觉压缩对于构建更高效、更鲁棒的视觉语言模型的潜力。