Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.

翻译：大型语言模型在多种复杂任务中表现出色。然而，在现实世界中实现通用推理（例如解决机器人问题）带来了具身化挑战。我们提出具身语言模型，将真实世界的连续传感器模态直接整合到语言模型中，从而建立词语与感知之间的关联。该具身语言模型的输入为多模态语句，交织了视觉、连续状态估计和文本输入编码。我们针对多个具身任务（包括顺序机器人操作规划、视觉问答和图像描述生成），以端到端方式训练这些编码，并与预训练的大型语言模型协同工作。评估表明，单一的大型具身多模态模型PaLM-E能够从多种观测模态出发，在多个具身形体上处理各类具身推理任务，并且展现出正迁移能力：该模型受益于互联网规模的语言、视觉及视觉-语言领域的多样化联合训练。我们的最大模型PaLM-E-562B（拥有562B参数）除了接受机器人任务训练外，还是一款视觉语言通才模型，在OK-VQA上达到业界领先性能，并随着规模扩大保持通才语言能力。