We report a controlled study investigating the effect of visual information (i.e., seeing the speaker) on spoken language comprehension. We compare the ERP signature (N400) associated with each word in audio-only and audio-visual presentations of the same verbal stimuli. We assess the extent to which surprisal measures (which quantify the predictability of words in their lexical context) are generated on the basis of different types of language models (specifically n-gram and Transformer models) that predict N400 responses for each word. Our results indicate that cognitive effort differs significantly between multimodal and unimodal settings. In addition, our findings suggest that while Transformer-based models, which have access to a larger lexical context, provide a better fit in the audio-only setting, 2-gram language models are more effective in the multimodal setting. This highlights the significant impact of local lexical context on cognitive processing in a multimodal environment.

翻译：我们报告了一项受控研究，旨在考察视觉信息（即观察说话者）对口语理解的影响。我们比较了同一言语刺激在纯听觉与视听呈现条件下，与每个词汇相关的ERP特征（N400）。我们评估了基于不同语言模型（具体为n-gram模型和Transformer模型）生成的惊异度指标（用于量化词汇在其词汇上下文中的可预测性）对每个词汇N400响应的预测程度。结果表明，多模态与单模态情境下的认知负荷存在显著差异。此外，我们的发现表明，尽管能访问更大词汇上下文的Transformer模型在纯听觉情境下拟合效果更佳，但2-gram语言模型在多模态情境下更为有效。这凸显了局部词汇上下文对多模态环境中认知加工的重要影响。