This research explores the integration of language embeddings for active learning in autonomous driving datasets, with a focus on novelty detection. Novelty arises from unexpected scenarios that autonomous vehicles struggle to navigate, necessitating higher-level reasoning abilities. Our proposed method employs language-based representations to identify novel scenes, emphasizing the dual purpose of safety takeover responses and active learning. The research presents a clustering experiment using Contrastive Language-Image Pretrained (CLIP) embeddings to organize datasets and detect novelties. We find that the proposed algorithm effectively isolates novel scenes from a collection of subsets derived from two real-world driving datasets, one vehicle-mounted and one infrastructure-mounted. From the generated clusters, we further present methods for generating textual explanations of elements which differentiate scenes classified as novel from other scenes in the data pool, presenting qualitative examples from the clustered results. Our results demonstrate the effectiveness of language-driven embeddings in identifying novel elements and generating explanations of data, and we further discuss potential applications in safe takeovers, data curation, and multi-task active learning.

翻译：本研究探索了语言嵌入在自动驾驶数据集主动学习中的应用，重点聚焦于新颖性检测。新颖性源于自动驾驶车辆难以应对的突发场景，这就需要更高层次的推理能力。我们提出的方法采用基于语言的表示来识别新颖场景，强调安全接管响应和主动学习的双重目标。本研究开展了一项聚类实验，利用对比语言-图像预训练（CLIP）嵌入来组织数据集并检测新颖性。我们发现，所提算法能有效从两个真实驾驶数据集（一个车载、一个路侧）衍生出的子集集合中分离出新颖场景。基于生成的聚类，我们进一步提出了生成文本解释的方法，用以区分数据池中被归类为新颖的场景与其他场景的元素，并展示了聚类结果的定性示例。我们的结果证明了语言驱动的嵌入在识别新颖元素和生成数据解释方面的有效性，并进一步讨论了在安全接管、数据整理和多任务主动学习中的潜在应用。