Embedding a language field in a 3D representation enables richer semantic understanding of spatial environments by linking geometry with descriptive meaning. This allows for a more intuitive human-computer interaction, enabling querying or editing scenes using natural language, and could potentially improve tasks like scene retrieval, navigation, and multimodal reasoning. While such capabilities could be transformative, in particular for large-scale scenes, we find that recent feature distillation approaches cannot effectively learn over massive Internet data due to challenges in semantic feature misalignment and inefficiency in memory and runtime. To this end, we propose a novel approach to address these challenges. First, we introduce extremely low-dimensional semantic bottleneck features as part of the underlying 3D Gaussian representation. These are processed by rendering and passing them through a multi-resolution, feature-based, hash encoder. This significantly improves efficiency both in runtime and GPU memory. Second, we introduce an Attenuated Downsampler module and propose several regularizations addressing the semantic misalignment of ground truth 2D features. We evaluate our method on the in-the-wild HolyScenes dataset and demonstrate that it surpasses existing approaches in both performance and efficiency.

翻译：在三维表示中嵌入语言场，通过将几何结构与描述性语义关联，能够实现对空间环境更丰富的语义理解。这为人机交互提供了更直观的方式，支持使用自然语言查询或编辑场景，并可能提升场景检索、导航及多模态推理等任务的性能。尽管此类能力具有变革性潜力，尤其适用于大规模场景，但我们发现，由于语义特征错位以及内存与运行时效率低下等挑战，近期的特征蒸馏方法难以有效利用海量互联网数据进行学习。为此，我们提出一种新颖方法以应对这些挑战。首先，我们在底层三维高斯表示中引入了极低维度的语义瓶颈特征，通过渲染并经由基于多分辨率特征的哈希编码器进行处理，显著提升了运行时效率与GPU内存利用率。其次，我们设计了衰减下采样模块，并提出多种正则化策略以解决真实二维特征中的语义错位问题。我们在真实场景数据集HolyScenes上评估了所提方法，结果表明其在性能与效率方面均优于现有方法。