Vision-Language-Action (VLA) models have demonstrated impressive capabilities in generalized robotic control; however, they remain notoriously brittle to linguistic perturbations. We identify a critical ``modality collapse'' phenomenon where strong visual priors overwhelm sparse linguistic signals, causing agents to overfit to specific instruction phrasings while ignoring the underlying semantic intent. To address this, we propose Residual Semantic Steering (RSS), a probabilistic framework that disentangles physical affordance from semantic execution. RSS introduces two theoretical innovations: (1) Monte Carlo Syntactic Integration, which approximates the true semantic posterior via dense, LLM-driven distributional expansion, and (2) Residual Affordance Steering, a dual-stream decoding mechanism that explicitly isolates the causal influence of language by subtracting the visual affordance prior. Theoretical analysis suggests that RSS effectively maximizes the mutual information between action and intent while suppressing visual distractors. Empirical results across diverse manipulation benchmarks demonstrate that RSS achieves state-of-the-art robustness, maintaining performance even under adversarial linguistic perturbations. We release our code at https://github.com/Doo-mon/RSS.

翻译：视觉-语言-动作（VLA）模型在通用机器人控制领域展现出卓越能力，但始终存在对语言扰动极度脆弱的固有问题。我们识别出关键的"模态坍缩"现象：强视觉先验压制了稀疏的语言信号，导致智能体过度拟合特定指令措辞而忽略潜在语义意图。为此，我们提出残差语义引导（RSS）——一种解耦物理可操作性语义执行的概率框架。RSS包含两项理论创新：（1）蒙特卡洛语法集成，通过大语言模型驱动的密集分布扩展近似真实语义后验；（2）残差可操作性引导，一种双流解码机制，通过消减视觉可操作性先验显式分离语言的因果影响。理论分析表明，RSS能有效最大化动作与意图间的互信息，同时抑制视觉干扰。在多种操作基准数据集上的实证结果显示，RSS实现了最先进的鲁棒性，即使在对抗性语言扰动下仍能保持性能。我们已开源代码：https://github.com/Doo-mon/RSS。