Humans use different modalities, such as speech, text, images, videos, etc., to communicate their intent and goals with teammates. For robots to become better assistants, we aim to endow them with the ability to follow instructions and understand tasks specified by their human partners. Most robotic policy learning methods have focused on one single modality of task specification while ignoring the rich cross-modal information. We present MUTEX, a unified approach to policy learning from multimodal task specifications. It trains a transformer-based architecture to facilitate cross-modal reasoning, combining masked modeling and cross-modal matching objectives in a two-stage training procedure. After training, MUTEX can follow a task specification in any of the six learned modalities (video demonstrations, goal images, text goal descriptions, text instructions, speech goal descriptions, and speech instructions) or a combination of them. We systematically evaluate the benefits of MUTEX in a newly designed dataset with 100 tasks in simulation and 50 tasks in the real world, annotated with multiple instances of task specifications in different modalities, and observe improved performance over methods trained specifically for any single modality. More information at https://ut-austin-rpl.github.io/MUTEX/

翻译：人类使用多种模态（如语音、文本、图像、视频等）来向队友传达意图和目标。为使机器人成为更优秀的助手，我们旨在赋予其遵循指令并理解人类伙伴指定任务的能力。大多数机器人策略学习方法仅关注单一模态的任务规范，忽视了丰富的跨模态信息。我们提出MUTEX，一种从多模态任务规范中学习策略的统一方法。该方法训练基于Transformer的架构以促进跨模态推理，结合掩码建模和跨模态匹配目标，采用两阶段训练流程。训练完成后，MUTEX能够遵循任意一种已学习的六种模态（视频演示、目标图像、文本目标描述、文本指令、语音目标描述和语音指令）或其组合的任务规范。我们在新设计的数据集上系统评估了MUTEX的优势，该数据集包含100个仿真任务和50个真实世界任务，并标注了多种模态的多个任务规范实例。实验观察到，相较于为单一模态专门训练的方法，MUTEX在性能上有所提升。更多信息请访问https://ut-austin-rpl.github.io/MUTEX/