Multimodal Large Language Models (MLLMs) achieve stronger visual understanding by scaling input fidelity, yet the resulting visual token growth makes jointly sustaining high spatial resolution and long temporal context prohibitive. We argue that the bottleneck lies not in how post-encoding representations are compressed but in the volume of pixels the encoder receives, and address it with ResAdapt, an Input-side adaptation framework that learns how much visual budget each frame should receive before encoding. ResAdapt couples a lightweight Allocator with an unchanged MLLM backbone, so the backbone retains its native visual-token interface while receiving an operator-transformed input. We formulate allocation as a contextual bandit and train the Allocator with Cost-Aware Policy Optimization (CAPO), which converts sparse rollout feedback into a stable accuracy-cost learning signal. Across budget-controlled video QA, temporal grounding, and image reasoning tasks, ResAdapt improves low-budget operating points and often lies on or near the efficiency-accuracy frontier, with the clearest gains on reasoning-intensive benchmarks under aggressive compression. Notably, ResAdapt supports up to 16x more frames at the same visual budget while delivering over 15% performance gain. Code is available at https://github.com/Xnhyacinth/ResAdapt.

翻译：多模态大语言模型通过提升输入保真度实现了更强的视觉理解能力，但由此产生的视觉标记数量激增使得同时维持高空间分辨率与长时序上下文成为不可能。我们认为瓶颈不在于编码后表示的压缩方式，而在于编码器接收的像素总量，并由此提出ResAdapt——一种在编码前学习各帧应获得多少视觉预算的输入侧自适应框架。ResAdapt将轻量级分配器与未改动的多模态大语言模型主干相结合，使主干在接收算子变换后的输入时保留其原生视觉标记接口。我们将分配问题形式化为上下文赌博机，并通过成本感知策略优化训练分配器，该算法将稀疏的轨迹反馈转化为稳定的精度-成本学习信号。在预算控制的视频问答、时序定位及图像推理任务中，ResAdapt提升了低预算工作点的性能，通常位于或接近效率-精度前沿曲线，在激进压缩条件下的推理密集型基准测试中获得了最显著的增益。值得注意的是，在相同视觉预算下ResAdapt支持多达16倍的帧数增幅，同时带来超过15%的性能提升。代码开源地址：https://github.com/Xnhyacinth/ResAdapt。