Recent end-to-end autonomous driving approaches have leveraged Vision-Language Models (VLMs) to enhance planning capabilities in complex driving scenarios. However, VLMs are inherently trained as generalist models, lacking specialized understanding of driving-specific reasoning in 3D space and time. When applied to autonomous driving, these models struggle to establish structured spatial-temporal representations that capture geometric relationships, scene context, and motion patterns critical for safe trajectory planning. To address these limitations, we propose SGDrive, a novel framework that explicitly structures the VLM's representation learning around driving-specific knowledge hierarchies. Built upon a pre-trained VLM backbone, SGDrive decomposes driving understanding into a scene-agent-goal hierarchy that mirrors human driving cognition: drivers first perceive the overall environment (scene context), then attend to safety-critical agents and their behaviors, and finally formulate short-term goals before executing actions. This hierarchical decomposition provides the structured spatial-temporal representation that generalist VLMs lack, integrating multi-level information into a compact yet comprehensive format for trajectory planning. Extensive experiments on the NAVSIM benchmark demonstrate that SGDrive achieves state-of-the-art performance among camera-only methods on both PDMS and EPDMS, validating the effectiveness of hierarchical knowledge structuring for adapting generalist VLMs to autonomous driving.

翻译：近期端到端自动驾驶方法利用视觉语言模型（VLM）来增强复杂驾驶场景中的规划能力。然而，视觉语言模型本质上作为通用模型进行训练，缺乏对三维时空领域中驾驶专用推理的专业理解。当应用于自动驾驶时，这些模型难以建立结构化的时空表征以捕捉几何关系、场景上下文及对安全轨迹规划至关重要的运动模式。为应对这些局限性，我们提出SGDrive——一种新颖的框架，该框架围绕驾驶专用知识层次显式地构建视觉语言模型的表征学习。基于预训练的视觉语言模型主干，SGDrive将驾驶理解分解为模拟人类驾驶认知的场景-智能体-目标层次结构：驾驶员首先感知整体环境（场景上下文），继而关注安全关键智能体及其行为，最终在执行动作前制定短期目标。这种层次化分解提供了通用视觉语言模型所缺乏的结构化时空表征，将多层级信息整合为紧凑而全面的格式以用于轨迹规划。在NAVSIM基准上的大量实验表明，SGDrive在仅使用摄像头的方法中于PDMS和EPDMS指标上均实现了最先进的性能，验证了层次化知识结构在将通用视觉语言模型适配至自动驾驶领域的有效性。