Foundation models have achieved remarkable results in 2D and language tasks like image segmentation, object detection, and visual-language understanding. However, their potential to enrich 3D scene representation learning is largely untapped due to the existence of the domain gap. In this work, we propose an innovative methodology called Bridge3D to address this gap by pre-training 3D models using features, semantic masks, and captions sourced from foundation models. Specifically, our method employs semantic masks from foundation models to guide the masking and reconstruction process for the masked autoencoder, enabling more focused attention on foreground representations. Moreover, we bridge the 3D-text gap at the scene level using image captioning foundation models, thereby facilitating scene-level knowledge distillation. We further extend this bridging effort by introducing an innovative object-level knowledge distillation method that harnesses highly accurate object-level masks and semantic text data from foundation models. Our methodology significantly surpasses the performance of existing state-of-the-art methods in 3D object detection and semantic segmentation tasks. For instance, on the ScanNet dataset, Bridge3D improves the baseline by a notable margin of 6.3%. Code will be available at: https://github.com/Zhimin-C/Bridge3D

翻译：基础模型在图像分割、目标检测及视觉语言理解等二维与语言任务中已取得显著成果。然而由于领域鸿沟的存在，其增强三维场景表征学习的潜力尚未被充分挖掘。本文提出名为Bridge3D的创新方法，通过利用基础模型提取的特征、语义掩码和文本描述预训练三维模型来弥合这一鸿沟。具体而言，本方法采用基础模型生成的语义掩码引导掩码自编码器的掩码与重建过程，从而增强对前景表征的专注力。同时，利用图像描述基础模型在场景层面弥合三维-文本差距，实现场景级知识蒸馏。进一步地，通过引入创新的目标级知识蒸馏方法——利用基础模型输出的高精度目标掩码与语义文本数据——扩展了上述弥合策略。本方法在三维目标检测与语义分割任务上显著超越现有最先进方法。以ScanNet数据集为例，Bridge3D将基准性能提升6.3%的显著幅度。代码开源地址：https://github.com/Zhimin-C/Bridge3D