Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed geometric clustering techniques. Building on the successes of recent Transformer-based methods for object detection and image segmentation, we propose the first Transformer-based approach for 3D semantic instance segmentation. We show that we can leverage generic Transformer building blocks to directly predict instance masks from 3D point clouds. In our model called Mask3D each object instance is represented as an instance query. Using Transformer decoders, the instance queries are learned by iteratively attending to point cloud features at multiple scales. Combined with point features, the instance queries directly yield all instance masks in parallel. Mask3D has several advantages over current state-of-the-art approaches, since it neither relies on (1) voting schemes which require hand-selected geometric properties (such as centers) nor (2) geometric grouping mechanisms requiring manually-tuned hyper-parameters (e.g. radii) and (3) enables a loss that directly optimizes instance masks. Mask3D sets a new state-of-the-art on ScanNet test (+6.2 mAP), S3DIS 6-fold (+10.1 mAP), STPLS3D (+11.2 mAP) and ScanNet200 test (+12.4 mAP).

翻译：现代三维语义实例分割方法主要依赖专门的投票机制，随后辅以精心设计的几何聚类技术。基于近期基于Transformer的目标检测与图像分割方法的成功，我们提出了首个基于Transformer的三维语义实例分割方法。研究表明，可直接利用通用Transformer构建模块从三维点云中预测实例掩码。在我们提出的Mask3D模型中，每个对象实例被表示为实例查询向量。通过Transformer解码器，这些实例查询向量通过迭代关注多尺度点云特征进行学习。结合点特征，实例查询向量能并行直接生成所有实例掩码。与现有最先进方法相比，Mask3D具有多项优势：既不依赖需要人工选择几何属性（如中心点）的投票方案，也不依赖需要手动调整超参数（如半径）的几何分组机制，且支持直接优化实例掩码的损失函数。Mask3D在ScanNet测试集（mAP提升6.2%）、S3DIS六折交叉验证（mAP提升10.1%）、STPLS3D（mAP提升11.2%）及ScanNet200测试集（mAP提升12.4%）上均创下新纪录。