Benchmarks for language-guided embodied agents typically assume text-based instructions, but deployed agents will encounter spoken instructions. While Automatic Speech Recognition (ASR) models can bridge the input gap, erroneous ASR transcripts can hurt the agents' ability to complete tasks. In this work, we propose training a multimodal ASR model to reduce errors in transcribing spoken instructions by considering the accompanying visual context. We train our model on a dataset of spoken instructions, synthesized from the ALFRED task completion dataset, where we simulate acoustic noise by systematically masking spoken words. We find that utilizing visual observations facilitates masked word recovery, with multimodal ASR models recovering up to 30% more masked words than unimodal baselines. We also find that a text-trained embodied agent successfully completes tasks more often by following transcribed instructions from multimodal ASR models. github.com/Cylumn/embodied-multimodal-asr

翻译：语言引导的具身智能体基准测试通常假设使用文本指令，但实际部署的智能体将接收语音指令。尽管自动语音识别（ASR）模型可以弥补输入方式的差异，但错误的ASR转录文本会损害智能体完成任务的能力。本研究提出通过考虑伴随的视觉上下文来训练多模态ASR模型，以减少口语指令转录中的错误。我们基于ALFRED任务完成数据集合成的口语指令数据集进行模型训练，通过系统性地屏蔽口语词汇来模拟声学噪声。研究发现，利用视觉观察有助于恢复被屏蔽的词汇，多模态ASR模型比单模态基线模型多恢复高达30%的屏蔽词汇。同时，文本训练的具身智能体在遵循多模态ASR模型转录的指令时，能够更频繁地成功完成任务。代码已开源：github.com/Cylumn/embodied-multimodal-asr