In the field of autonomous vehicles (AVs), accurately discerning commander intent and executing linguistic commands within a visual context presents a significant challenge. This paper introduces a sophisticated encoder-decoder framework, developed to address visual grounding in AVs.Our Context-Aware Visual Grounding (CAVG) model is an advanced system that integrates five core encoders-Text, Image, Context, and Cross-Modal-with a Multimodal decoder. This integration enables the CAVG model to adeptly capture contextual semantics and to learn human emotional features, augmented by state-of-the-art Large Language Models (LLMs) including GPT-4. The architecture of CAVG is reinforced by the implementation of multi-head cross-modal attention mechanisms and a Region-Specific Dynamic (RSD) layer for attention modulation. This architectural design enables the model to efficiently process and interpret a range of cross-modal inputs, yielding a comprehensive understanding of the correlation between verbal commands and corresponding visual scenes. Empirical evaluations on the Talk2Car dataset, a real-world benchmark, demonstrate that CAVG establishes new standards in prediction accuracy and operational efficiency. Notably, the model exhibits exceptional performance even with limited training data, ranging from 50% to 75% of the full dataset. This feature highlights its effectiveness and potential for deployment in practical AV applications. Moreover, CAVG has shown remarkable robustness and adaptability in challenging scenarios, including long-text command interpretation, low-light conditions, ambiguous command contexts, inclement weather conditions, and densely populated urban environments. The code for the proposed model is available at our Github.

翻译：在自动驾驶车辆领域，准确识别指挥者意图并在视觉场景中执行语言指令仍是一项重大挑战。本文提出了一种先进的编码器-解码器框架，旨在解决自动驾驶车辆中的视觉接地问题。所提出的上下文感知视觉接地（CAVG）模型是一个集成五大核心编码器（文本、图像、上下文、跨模态）与多模态解码器的先进系统。该集成使CAVG模型能够精确捕捉上下文语义，并学习人类情感特征，同时得到包括GPT-4在内的最先进大型语言模型（LLMs）的增强。CAVG架构通过引入多头跨模态注意力机制和区域特定动态（RSD）注意力调制层得到强化。该架构设计使模型能够高效处理并解读多种跨模态输入，从而全面理解语言指令与对应视觉场景之间的关联。在真实世界基准数据集Talk2Car上的实验评估表明，CAVG在预测精度和运行效率上树立了新标杆。值得注意的是，即使仅使用完整数据集的50%至75%进行训练，该模型仍展现出卓越性能，凸显了其在实际自动驾驶应用中部署的潜力与有效性。此外，CAVG在长文本指令理解、弱光环境、模糊指令语境、恶劣天气条件及高密度城市区域等挑战性场景中，表现出显著的鲁棒性和适应性。本文所提模型的代码已发布于GitHub。