Humans describe the physical world using natural language to refer to specific 3D locations based on a vast range of properties: visual appearance, semantics, abstract associations, or actionable affordances. In this work we propose Language Embedded Radiance Fields (LERFs), a method for grounding language embeddings from off-the-shelf models like CLIP into NeRF, which enable these types of open-ended language queries in 3D. LERF learns a dense, multi-scale language field inside NeRF by volume rendering CLIP embeddings along training rays, supervising these embeddings across training views to provide multi-view consistency and smooth the underlying language field. After optimization, LERF can extract 3D relevancy maps for a broad range of language prompts interactively in real-time, which has potential use cases in robotics, understanding vision-language models, and interacting with 3D scenes. LERF enables pixel-aligned, zero-shot queries on the distilled 3D CLIP embeddings without relying on region proposals or masks, supporting long-tail open-vocabulary queries hierarchically across the volume. The project website can be found at https://lerf.io .

翻译：人类通过自然语言描述物理世界，依据视觉外观、语义、抽象关联或可操作功能等广泛属性，指代特定三维位置。本文提出语言嵌入辐射场（LERF）方法，将预训练模型（如CLIP）的语言嵌入注入NeRF，从而在三维场景中实现此类开放式语言查询。LERF通过沿训练射线体渲染CLIP嵌入，在NeRF内部学习密集的多尺度语言场，并跨训练视角监督这些嵌入以保持多视图一致性并平滑底层语言场。优化后，LERF可实时交互式提取针对多种语言提示的三维相关性图，在机器人学、视觉-语言模型理解及三维场景交互中具有潜在应用价值。LERF能够对蒸馏后的三维CLIP嵌入执行像素对齐的零样本查询，无需依赖区域提议或掩码，支持跨体层级的长尾开放词汇查询。项目网站详见https://lerf.io。