This paper introduces a novel method for open-vocabulary 3D scene querying in autonomous driving by combining Language Embedded 3D Gaussians with Large Language Models (LLMs). We propose utilizing LLMs to generate both contextually canonical phrases and helping positive words for enhanced segmentation and scene interpretation. Our method leverages GPT-3.5 Turbo as an expert model to create a high-quality text dataset, which we then use to fine-tune smaller, more efficient LLMs for on-device deployment. Our comprehensive evaluation on the WayveScenes101 dataset demonstrates that LLM-guided segmentation significantly outperforms traditional approaches based on predefined canonical phrases. Notably, our fine-tuned smaller models achieve performance comparable to larger expert models while maintaining faster inference times. Through ablation studies, we discover that the effectiveness of helping positive words correlates with model scale, with larger models better equipped to leverage additional semantic information. This work represents a significant advancement towards more efficient, context-aware autonomous driving systems, effectively bridging 3D scene representation with high-level semantic querying while maintaining practical deployment considerations.

翻译：本文提出了一种结合语言嵌入三维高斯与大型语言模型（LLMs）的新型自动驾驶开集词汇三维场景查询方法。我们提出利用LLMs生成上下文规范短语及辅助正向词汇，以增强分割效果与场景理解。该方法采用GPT-3.5 Turbo作为专家模型构建高质量文本数据集，并基于此微调更轻量高效的LLMs以实现端侧部署。在WayveScenes101数据集上的综合评估表明，LLM引导的分割方法显著优于基于预定义规范短语的传统方案。值得注意的是，经微调的轻量化模型在保持更快推理速度的同时，达到了与大型专家模型相当的性能。通过消融实验发现，辅助正向词汇的有效性与模型规模相关，更大规模的模型能更充分地利用附加语义信息。本工作代表了向更高效、具上下文感知能力的自动驾驶系统迈出的重要进展，在保持实际部署可行性的同时，有效实现了三维场景表征与高层语义查询的有机融合。