Multimodal Mixture-of-Experts (MoE) models have achieved remarkable performance on vision-language tasks. However, we identify a puzzling phenomenon termed Seeing but Not Thinking: models accurately perceive image content yet fail in subsequent reasoning, while correctly solving identical problems presented as pure text. Through systematic analysis, we first verify that cross-modal semantic sharing exists in MoE architectures, ruling out semantic alignment failure as the sole explanation. We then reveal that visual experts and domain experts exhibit layer-wise separation, with image inputs inducing significant routing divergence from text inputs in middle layers where domain experts concentrate. Based on these findings, we propose the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts. To validate this hypothesis, we design a routing-guided intervention method that enhances domain expert activation. Experiments on three multimodal MoE models across six benchmarks demonstrate consistent improvements, with gains of up to 3.17% on complex visual reasoning tasks. Our analysis further reveals that domain expert identification locates cognitive functions rather than sample-specific solutions, enabling effective transfer across tasks with different information structures.
翻译:多模态混合专家(MoE)模型在视觉-语言任务中取得了显著性能。然而,我们识别出一个被称为“看见却不想”的费解现象:模型能准确感知图像内容,却在后续推理中失败,而处理相同问题的纯文本版本时却能正确解答。通过系统分析,我们首先验证了MoE架构中跨模态语义共享的存在性,排除了语义对齐失败作为唯一解释的可能性。进一步揭示视觉专家与领域专家在层级上存在分离,图像输入在中间层(领域专家集中区域)引发与文本输入显著不同的路由分歧。基于这些发现,我们提出“路由干扰”假说:在处理视觉输入时,路由机制未能充分激活任务相关推理专家。为验证该假说,我们设计了一种路由引导的干预方法,用于增强领域专家激活。在六个基准测试中对三个多模态MoE模型的实验显示出一致性改进,在复杂视觉推理任务上的性能提升高达3.17%。我们的分析进一步表明,领域专家识别定位的是认知功能而非样本特定解,从而能够实现跨不同信息结构任务的有效迁移。