Multimodal Large Language Models (MLLMs) achieve stronger visual understanding by scaling input fidelity, yet the resulting visual token growth makes jointly sustaining high spatial resolution and long temporal context prohibitive. We argue that the bottleneck lies not in how post-encoding representations are compressed but in the volume of pixels the encoder receives, and address it with ResAdapt, an Input-side adaptation framework that learns how much visual budget each frame should receive before encoding. ResAdapt couples a lightweight Allocator with an unchanged MLLM backbone, so the backbone retains its native visual-token interface while receiving an operator-transformed input. We formulate allocation as a contextual bandit and train the Allocator with Cost-Aware Policy Optimization (CAPO), which converts sparse rollout feedback into a stable accuracy-cost learning signal. Across budget-controlled video QA, temporal grounding, and image reasoning tasks, ResAdapt improves low-budget operating points and often lies on or near the efficiency-accuracy frontier, with the clearest gains on reasoning-intensive benchmarks under aggressive compression. Notably, ResAdapt supports up to 16x more frames at the same visual budget while delivering over 15% performance gain. Code is available at https://github.com/Xnhyacinth/ResAdapt.

翻译：[translated abstract in Chinese] 多模态大语言模型通过提升输入保真度实现了更强的视觉理解能力，然而由此带来的视觉标记数量增长使得同时维持高空间分辨率与长时间上下文变得不可行。我们认为瓶颈并非在于编码后表征的压缩方式，而在于编码器接收到的像素总量，并据此提出ResAdapt——一种编码前的输入侧自适应框架，能够学习在编码前为每一帧分配多少视觉预算。ResAdapt将轻量级的分配器与未经修改的多模态大语言模型主架构相结合，使得主架构在接收经过算子变换的输入时，仍能保留其原有的视觉标记接口。我们将分配问题形式化为上下文赌博机，并采用成本感知策略优化（CAPO）训练分配器，该方法能将稀疏的 rollout 反馈转化为稳定的准确率-成本学习信号。在预算受限的视频问答、时间定位及图像推理任务中，ResAdapt 提升了低预算下的性能表现点，且其性能通常位于或接近效率-准确率前沿，尤其在推理密集型基准测试中进行高倍压缩时，性能提升最为显著。值得注意的是，在相同视觉预算下，ResAdapt 能支持最多16倍的帧数，同时带来超过15%的性能增益。代码开源于 https://github.com/Xnhyacinth/ResAdapt。