We present a Multimodal Interlaced Transformer (MIT) that jointly considers 2D and 3D data for weakly supervised point cloud segmentation. Research studies have shown that 2D and 3D features are complementary for point cloud segmentation. However, existing methods require extra 2D annotations to achieve 2D-3D information fusion. Considering the high annotation cost of point clouds, effective 2D and 3D feature fusion based on weakly supervised learning is in great demand. To this end, we propose a transformer model with two encoders and one decoder for weakly supervised point cloud segmentation using only scene-level class tags. Specifically, the two encoders compute the self-attended features for 3D point clouds and 2D multi-view images, respectively. The decoder implements interlaced 2D-3D cross-attention and carries out implicit 2D and 3D feature fusion. We alternately switch the roles of queries and key-value pairs in the decoder layers. It turns out that the 2D and 3D features are iteratively enriched by each other. Experiments show that it performs favorably against existing weakly supervised point cloud segmentation methods by a large margin on the S3DIS and ScanNet benchmarks. The project page will be available at https://jimmy15923.github.io/mit_web/.

翻译：我们提出了一种多模态交错Transformer（MIT），该模型联合考虑2D和3D数据以实现弱监督点云分割。研究表明，2D和3D特征在点云分割中具有互补性。然而，现有方法需要额外的2D标注才能实现2D-3D信息融合。考虑到点云的高标注成本，基于弱监督学习的有效2D与3D特征融合需求迫切。为此，我们提出一种包含两个编码器和一个解码器的Transformer模型，仅利用场景级类别标签实现弱监督点云分割。具体而言，两个编码器分别计算3D点云和2D多视角图像的自注意力特征；解码器则实现交错的2D-3D交叉注意力机制，并隐式完成2D与3D特征融合。我们在解码器层中交替切换查询与键值对的作用。实验表明，2D和3D特征通过相互迭代得到丰富增强。在S3DIS和ScanNet基准数据集上，该方法相比现有弱监督点云分割方法取得了显著性能提升。项目页面将发布于https://jimmy15923.github.io/mit_web/。