This paper introduces OpenGaussian, a method based on 3D Gaussian Splatting (3DGS) capable of 3D point-level open vocabulary understanding. Our primary motivation stems from observing that existing 3DGS-based open vocabulary methods mainly focus on 2D pixel-level parsing. These methods struggle with 3D point-level tasks due to weak feature expressiveness and inaccurate 2D-3D feature associations. To ensure robust feature presentation and 3D point-level understanding, we first employ SAM masks without cross-frame associations to train instance features with 3D consistency. These features exhibit both intra-object consistency and inter-object distinction. Then, we propose a two-stage codebook to discretize these features from coarse to fine levels. At the coarse level, we consider the positional information of 3D points to achieve location-based clustering, which is then refined at the fine level. Finally, we introduce an instance-level 3D-2D feature association method that links 3D points to 2D masks, which are further associated with 2D CLIP features. Extensive experiments, including open vocabulary-based 3D object selection, 3D point cloud understanding, click-based 3D object selection, and ablation studies, demonstrate the effectiveness of our proposed method. The source code is available at our project page: https://3d-aigc.github.io/OpenGaussian

翻译：本文提出OpenGaussian，一种基于3D高斯泼溅（3DGS）且能实现3D点级开放词汇理解的方法。我们的主要动机源于观察到现有基于3DGS的开放词汇方法主要关注2D像素级解析。由于特征表达能力较弱以及2D-3D特征关联不准确，这些方法在3D点级任务上存在困难。为确保鲁棒的特征表示与3D点级理解，我们首先利用无跨帧关联的SAM掩码来训练具有3D一致性的实例特征。这些特征同时表现出对象内部一致性与对象间区分性。随后，我们提出一种两阶段码本，将这些特征从粗粒度到细粒度进行离散化。在粗粒度层面，我们考虑3D点的位置信息以实现基于位置的聚类，该聚类结果在细粒度层面进一步优化。最后，我们提出一种实例级3D-2D特征关联方法，将3D点与2D掩码相关联，这些掩码再进一步与2D CLIP特征建立联系。大量实验，包括基于开放词汇的3D对象选择、3D点云理解、基于点击的3D对象选择以及消融研究，均证明了所提方法的有效性。源代码可在项目页面获取：https://3d-aigc.github.io/OpenGaussian