End-to-end manipulation policies, combined with web-scale pretrained Vision-Language Models (VLMs), show the promise for generalizable and dexterous robotic manipulation. However, they inherit two key limitations from 2D foundation models: 1) the reliance on 2D RGB inputs that ignores the intrinsically 3D nature of manipulation; and 2) the lack of spatial 3D alignment between input-output spaces as well as across diverse robot embodiments, camera setups, and trajectory datasets. In this paper, we present a series of contributions to address these issues. First, we introduce aligned vertex map and vertex spectrum -- a pixel-wise 3D representation that elevates 2D visual inputs to 3D, using camera calibration and optional depth. This novel input representation marries 3D awareness with the generalization of 2D large VLMs. Then, we propose to align the inputs and outputs of manipulation policies by expressing per-pixel 3D information of each camera view and robot actions to a shared coordinate. Based on this, we designate a canonical Bird's-Eye-View (BEV) alignment frame and innovatively propose to construct BEV images, producing a view-invariant representation robust to camera pose variations. To enable training and evaluation at scale, we develop a comprehensive data processing pipeline to perform such alignments; we also introduce a novel temporal alignment scheme for trajectories across diverse robots, human operators, and datasets. These contributions collectively mitigate input and output spatial-temporal misalignments, improving the consistency and generalization for real-world manipulation. Pretrained checkpoint, source code and data processing pipeline are available in https://hnuzhy.github.io/projects/Dex-BEV.

翻译：端到端操作策略结合大规模预训练的视觉-语言模型（VLMs）展现出实现通用化灵巧机器人操作的潜力。然而，它们继承了二维基础模型的两个关键局限性：1）依赖于二维RGB输入，忽略了操作本身固有的三维特性；2）输入-输出空间之间以及不同机器人形态、相机设置和轨迹数据集之间缺乏空间三维对齐。本文提出一系列贡献来解决这些问题。首先，我们引入对齐顶点图和顶点光谱——一种像素级的三维表示，利用相机标定和可选深度信息将二维视觉输入提升至三维。这种新颖的输入表示将三维感知能力与二维大规模VLMs的泛化能力相结合。然后，我们提出通过将每个相机视角的像素级三维信息和机器人动作表达至共享坐标系，来对齐操作策略的输入和输出。基于此，我们指定标准鸟瞰图（BEV）对齐框架，并创新性地提出构建BEV图像，生成对相机姿态变化鲁棒的视角不变表示。为支持大规模训练和评估，我们开发了全面的数据处理流程来实现此类对齐；同时针对跨不同机器人、人类操作员和数据集的轨迹提出新颖的时间对齐方案。这些贡献共同缓解了输入与输出的时空错位问题，提升了真实世界操作的一致性和泛化能力。预训练检查点、源代码及数据处理流程可在https://hnuzhy.github.io/projects/Dex-BEV获取。