Spoken language instructions are ubiquitous in agent collaboration. However, in human-robot collaboration, recognition accuracy for human speech is often influenced by various speech and environmental factors, such as background noise, the speaker's accents, and mispronunciation. When faced with noisy or unfamiliar auditory inputs, humans use context and prior knowledge to disambiguate the stimulus and take pragmatic actions, a process referred to as top-down processing in cognitive science. We present a cognitively inspired model, Speech Instruction Following through Theory of Mind (SIFToM), to enable robots to pragmatically follow human instructions under diverse speech conditions by inferring the human's goal and joint plan as prior for speech perception and understanding. We test SIFToM in simulated home experiments (VirtualHome 2). Results show that the SIFToM model outperforms state-of-the-art speech and language models, approaching human-level accuracy on challenging speech instruction following tasks. We then demonstrate its ability at the task planning level on a mobile manipulator for breakfast preparation tasks.

翻译：口语指令在智能体协作中无处不在。然而，在人机协作中，对人类语音的识别准确率常常受到各种语音和环境因素的影响，例如背景噪声、说话者的口音和发音错误。面对嘈杂或不熟悉的听觉输入时，人类会利用上下文和先验知识来消除刺激的歧义并采取符合语用目的的行动，这一过程在认知科学中被称为自上而下处理。我们提出了一种受认知启发的模型——通过心智理论的语音指令跟随模型（SIFToM），使机器人能够通过推断人类的目标和联合计划作为语音感知与理解的先验，从而在各种语音条件下语用地遵循人类指令。我们在模拟家庭环境（VirtualHome 2）中测试了SIFToM。结果表明，SIFToM模型在具有挑战性的语音指令跟随任务上超越了当前最先进的语音和语言模型，其准确率接近人类水平。随后，我们在一个移动机械臂上执行早餐准备任务，展示了其在任务规划层面的能力。