State-of-the-art models on contemporary 3D perception benchmarks like ScanNet consume and label dataset-provided 3D point clouds, obtained through post processing of sensed multiview RGB-D images. They are typically trained in-domain, forego large-scale 2D pre-training and outperform alternatives that featurize the posed RGB-D multiview images instead. The gap in performance between methods that consume posed images versus post-processed 3D point clouds has fueled the belief that 2D and 3D perception require distinct model architectures. In this paper, we challenge this view and propose ODIN (Omni-Dimensional INstance segmentation), a model that can segment and label both 2D RGB images and 3D point clouds, using a transformer architecture that alternates between 2D within-view and 3D cross-view information fusion. Our model differentiates 2D and 3D feature operations through the positional encodings of the tokens involved, which capture pixel coordinates for 2D patch tokens and 3D coordinates for 3D feature tokens. ODIN achieves state-of-the-art performance on ScanNet200, Matterport3D and AI2THOR 3D instance segmentation benchmarks, and competitive performance on ScanNet, S3DIS and COCO. It outperforms all previous works by a wide margin when the sensed 3D point cloud is used in place of the point cloud sampled from 3D mesh. When used as the 3D perception engine in an instructable embodied agent architecture, it sets a new state-of-the-art on the TEACh action-from-dialogue benchmark. Our code and checkpoints can be found at the project website: https://odin-seg.github.io.

翻译：当前诸如ScanNet等3D感知基准上的最优模型，会使用并标注数据集提供的3D点云，这些点云通过对多视角RGB-D图像进行后处理获得。这类模型通常采用领域内训练，舍弃大规模2D预训练，且性能优于对带位姿的RGB-D多视角图像进行特征提取的替代方案。以带位姿图像为输入的方法与以后处理3D点云为输入的方法之间的性能差距，催生了"2D与3D感知需要不同模型架构"的认知。本文对这一观点提出挑战，并提出ODIN（全方位实例分割）模型——该模型采用交替进行2D视图内融合与3D跨视图信息融合的Transformer架构，可同时对2D RGB图像和3D点云进行分割与标注。我们的模型通过token的位置编码来区分2D与3D特征操作：2D块token捕获像素坐标，3D特征token捕获三维坐标。ODIN在ScanNet200、Matterport3D和AI2THOR 3D实例分割基准上达到最优性能，在ScanNet、S3DIS和COCO上亦具有竞争力。当使用传感器采集的3D点云替代从3D网格采样的点云时，该方法以显著优势超越所有先前工作。作为可指令具体化智能体架构的3D感知引擎时，该方法在TEACh对话动作基准上创下新纪录。我们的代码与模型检查点可在项目网站获取：https://odin-seg.github.io。