Learning to build 3D scene graphs is essential for real-world perception in a structured and rich fashion. However, previous 3D scene graph generation methods utilize a fully supervised learning manner and require a large amount of entity-level annotation data of objects and relations, which is extremely resource-consuming and tedious to obtain. To tackle this problem, we propose 3D-VLAP, a weakly-supervised 3D scene graph generation method via Visual-Linguistic Assisted Pseudo-labeling. Specifically, our 3D-VLAP exploits the superior ability of current large-scale visual-linguistic models to align the semantics between texts and 2D images, as well as the naturally existing correspondences between 2D images and 3D point clouds, and thus implicitly constructs correspondences between texts and 3D point clouds. First, we establish the positional correspondence from 3D point clouds to 2D images via camera intrinsic and extrinsic parameters, thereby achieving alignment of 3D point clouds and 2D images. Subsequently, a large-scale cross-modal visual-linguistic model is employed to indirectly align 3D instances with the textual category labels of objects by matching 2D images with object category labels. The pseudo labels for objects and relations are then produced for 3D-VLAP model training by calculating the similarity between visual embeddings and textual category embeddings of objects and relations encoded by the visual-linguistic model, respectively. Ultimately, we design an edge self-attention based graph neural network to generate scene graphs of 3D point cloud scenes. Extensive experiments demonstrate that our 3D-VLAP achieves comparable results with current advanced fully supervised methods, meanwhile significantly alleviating the pressure of data annotation.

翻译：学习构建三维场景图对于以结构化和丰富的方式进行真实世界感知至关重要。然而，以往的三维场景图生成方法采用全监督学习方式，需要大量关于物体和关系的实体级标注数据，这极其耗费资源且繁琐难获。为解决此问题，我们提出3D-VLAP，一种通过视觉-语言辅助伪标记的弱监督三维场景图生成方法。具体而言，我们的3D-VLAP利用当前大规模视觉-语言模型对齐文本与二维图像语义的卓越能力，以及二维图像与三维点云之间天然存在的对应关系，从而隐式构建文本与三维点云之间的对应。首先，通过相机内参和外参建立三维点云到二维图像的位置对应，从而实现三维点云与二维图像的对齐。随后，利用大规模跨模态视觉-语言模型，通过匹配二维图像与物体类别标签，间接将三维实例与物体文本类别标签对齐。接着，通过计算视觉-语言模型编码的物体和关系的视觉嵌入与文本类别嵌入之间的相似度，为3D-VLAP模型训练生成物体和关系的伪标签。最终，我们设计了一种基于边缘自注意力的图神经网络，以生成三维点云场景的场景图。大量实验表明，我们的3D-VLAP在取得与当前先进全监督方法可比结果的同时，显著减轻了数据标注压力。