Benchmarks for language-guided embodied agents typically assume text-based instructions, but deployed agents will encounter spoken instructions. While Automatic Speech Recognition (ASR) models can bridge the input gap, erroneous ASR transcripts can hurt the agents' ability to complete tasks. In this work, we propose training a multimodal ASR model to reduce errors in transcribing spoken instructions by considering the accompanying visual context. We train our model on a dataset of spoken instructions, synthesized from the ALFRED task completion dataset, where we simulate acoustic noise by systematically masking spoken words. We find that utilizing visual observations facilitates masked word recovery, with multimodal ASR models recovering up to 30% more masked words than unimodal baselines. We also find that a text-trained embodied agent successfully completes tasks more often by following transcribed instructions from multimodal ASR models. github.com/Cylumn/embodied-multimodal-asr

翻译：语言引导具身智能体的基准测试通常假设指令以文本形式呈现，但实际部署的智能体将接收语音指令。虽然自动语音识别（ASR）模型可以填补输入层面的差距，但错误的ASR转录文本会损害智能体完成任务的能力。在本工作中，我们提出训练一种多模态ASR模型，通过考虑伴随的视觉上下文来减少口语指令转录中的错误。我们在基于ALFRED任务完成数据集合成的口语指令数据集上训练该模型，其中通过系统性地掩蔽口语词汇来模拟声学噪声。研究发现，利用视觉观察有助于恢复被掩蔽的词汇，多模态ASR模型相比单模态基线能够多恢复高达30%的被掩蔽词汇。我们还发现，文本训练的具身智能体在遵循多模态ASR模型转录的指令时，能够更频繁地成功完成任务。github.com/Cylumn/embodied-multimodal-asr