Multimodal large language models (MLLMs) achieve remarkable progress in cross-modal perception and reasoning, yet a fundamental question remains unresolved: should the vision encoder be fine-tuned or frozen? Despite the success of models such as LLaVA and Qwen-VL, inconsistent design choices and heterogeneous training setups hinder a unified understanding of visual fine-tuning (VFT) in MLLMs. Through a configuration-aligned benchmark, we find that existing VFT methods fail to consistently outperform the frozen baseline across multimodal tasks. Our analysis suggests that this instability arises from visual preference conflicts, where the context-agnostic nature of vision encoders induces divergent parameter updates under diverse multimodal context. To address this issue, we propose the Context-aware Visual Fine-tuning (CoVFT) framework, which explicitly incorporates multimodal context into visual adaptation. By integrating a Context Vector Extraction (CVE) and a Contextual Mixture-of-Experts (CoMoE) module, CoVFT decomposes conflicting optimization signals and enables stable, context-sensitive visual updates. Extensive experiments on 12 multimodal benchmarks demonstrate that CoVFT achieves state-of-the-art performance with superior stability. Notably, fine-tuning a 7B MLLM with CoVFT surpasses the average performance of its 13B counterpart, revealing substantial untapped potential in visual encoder optimization within MLLMs.

翻译：多模态大语言模型在跨模态感知与推理方面取得了显著进展，但一个根本性问题仍未得到解答：视觉编码器应进行微调还是冻结？尽管LLaVA、Qwen-VL等模型取得了成功，但设计选择的不一致性和训练配置的异构性阻碍了对多模态大语言模型中视觉微调的统一认识。通过配置对齐的基准测试，本研究发现现有视觉微调方法在多模态任务中无法持续优于冻结基线模型。分析表明，这种不稳定性源于视觉偏好冲突——视觉编码器的上下文无关特性会在不同多模态语境下引发参数更新的分歧。为解决该问题，本工作提出上下文感知视觉微调（CoVFT）框架，通过显式将多模态上下文融入视觉适配过程。通过集成上下文向量提取（CVE）与上下文专家混合（CoMoE）模块，CoVFT分解冲突的优化信号，实现稳定且上下文敏感的视觉参数更新。在12个多模态基准测试上的广泛实验表明，CoVFT以卓越的稳定性达到最优性能。值得注意的是，采用CoVFT微调7B参数量的多模态大语言模型，其平均表现已超越13B参数量的对应模型，揭示了多模态大语言模型中视觉编码器优化尚未开发的巨大潜力。