Multimodal foundation models integrate heterogeneous signals across modalities, yet it remains poorly understood how their predictions depend on specific internal feature groups and whether such reliance can be deliberately controlled. Existing studies of shortcut and spurious behavior largely rely on post hoc analyses or feature removal, offering limited insight into whether reliance can be modulated without altering task semantics. We introduce FiLoRA (Focus-and-Ignore LoRA), an instruction-conditioned, parameter-efficient adaptation framework that enables explicit control over internal feature reliance while keeping the predictive objective fixed. FiLoRA decomposes adaptation into feature group-aligned LoRA modules and applies instruction-conditioned gating, allowing natural language instructions to act as computation-level control signals rather than task redefinitions. Across text--image and audio--visual benchmarks, we show that instruction-conditioned gating induces consistent and causal shifts in internal computation, selectively amplifying or suppressing core and spurious feature groups without modifying the label space or training objective. Further analyses demonstrate that FiLoRA yields improved robustness under spurious feature interventions, revealing a principled mechanism to regulate reliance beyond correlation-driven learning.

翻译：多模态基础模型整合了跨模态的异构信号，然而其预测如何依赖于特定的内部特征组，以及这种依赖是否可以被有意控制，目前仍缺乏深入理解。现有关于捷径行为和伪相关特征的研究主要依赖于事后分析或特征移除方法，对于能否在不改变任务语义的前提下调节特征依赖，提供的见解有限。本文提出FiLoRA（聚焦-忽略低秩自适应），一种基于指令调节的参数高效适配框架，能够在保持预测目标不变的前提下，实现对内部特征依赖的显式控制。FiLoRA将适配过程分解为与特征组对齐的LoRA模块，并应用指令调节门控机制，使自然语言指令能够作为计算层面的控制信号，而非任务重定义手段。在文本-图像和音频-视觉基准测试中，我们证明指令调节门控能够引发内部计算的一致性和因果性转变，选择性地增强或抑制核心特征组与伪相关特征组，而无需修改标签空间或训练目标。进一步分析表明，FiLoRA在伪相关特征干预下展现出更强的鲁棒性，为超越相关性驱动学习的特征依赖调控提供了原理性机制。