In this paper, we introduce a method for unifying language, action, and state information in a shared embedding space to facilitate a range of downstream tasks in robot learning. Our method, Contrastive Language, Action, and State Pre-training (CLASP), extends the CLIP formulation by incorporating distributional learning, capturing the inherent complexities and one-to-many relationships in behaviour-text alignment. By employing distributional outputs for both text and behaviour encoders, our model effectively associates diverse textual commands with a single behaviour and vice-versa. We demonstrate the utility of our method for the following downstream tasks: zero-shot text-behaviour retrieval, captioning unseen robot behaviours, and learning a behaviour prior for language-conditioned reinforcement learning. Our distributional encoders exhibit superior retrieval and captioning performance on unseen datasets, and the ability to generate meaningful exploratory behaviours from textual commands, capturing the intricate relationships between language, action, and state. This work represents an initial step towards developing a unified pre-trained model for robotics, with the potential to generalise to a broad range of downstream tasks.

翻译：本文提出了一种在共享嵌入空间中统一语言、动作与状态信息的方法，以促进机器人学习中的一系列下游任务。该方法名为“对比语言、动作与状态预训练”（CLASP），通过引入分布学习扩展了CLIP（对比语言-图像预训练）框架，捕捉行为-文本对齐中的内在复杂性与一对多关系。通过为文本和行为编码器采用分布输出，我们的模型能够有效关联多样化的文本指令与单一行为，反之亦然。我们展示了该方法在以下下游任务中的效用：零样本文本-行为检索、对未见机器人行为的描述，以及为基于语言的条件强化学习学习行为先验。我们的分布编码器在未见数据集上展现出更优的检索与描述性能，并能从文本指令中生成有意义的探索性行为，捕捉语言、动作与状态之间的复杂关系。本研究是迈向开发统一机器人预训练模型的初步步骤，有望推广至广泛的下游任务。