Robots rely heavily on sensors, especially RGB and depth cameras, to perceive and interact with the world. RGB cameras record 2D images with rich semantic information while missing precise spatial information. On the other side, depth cameras offer critical 3D geometry data but capture limited semantics. Therefore, integrating both modalities is crucial for learning representations for robotic perception and control. However, current research predominantly focuses on only one of these modalities, neglecting the benefits of incorporating both. To this end, we present Semantic-Geometric Representation (SGR), a universal perception module for robotics that leverages the rich semantic information of large-scale pre-trained 2D models and inherits the merits of 3D spatial reasoning. Our experiments demonstrate that SGR empowers the agent to successfully complete a diverse range of simulated and real-world robotic manipulation tasks, outperforming state-of-the-art methods significantly in both single-task and multi-task settings. Furthermore, SGR possesses the unique capability to generalize to novel semantic attributes, setting it apart from the other methods.

翻译：机器人严重依赖传感器，尤其是RGB摄像头和深度摄像头来感知世界并进行交互。RGB摄像头能够记录富含语义信息的二维图像，但缺乏精确的空间信息；而深度摄像头可提供关键的三维几何数据，但仅能捕获有限的语义信息。因此，融合这两种模态对于学习机器人感知与控制的表示至关重要。然而，当前研究主要仅关注其中一种模态，忽视了融合两者的优势。为此，我们提出语义-几何表示（SGR），一种面向机器人的通用感知模块，它利用大规模预训练二维模型的丰富语义信息，同时继承三维空间推理的优势。实验表明，SGR使智能体能够成功完成模拟和真实环境中多种多样的机器人操作任务，在单任务和多任务场景下均显著优于现有最先进方法。此外，SGR具备泛化到新型语义属性的独特能力，这使其与其他方法形成显著区别。