Generalization under distribution shift remains a central bottleneck for closed-loop autonomous driving. Although simulators like CARLA enable safe and scalable testing, existing benchmarks rarely measure true generalization: they typically reuse training scenarios at test time. Success can therefore reflect memorization rather than robust driving behavior. We introduce Fail2Drive, the first paired-route benchmark for closed-loop generalization in CARLA, with 200 routes and 17 new scenario classes spanning appearance, layout, behavioral, and robustness shifts. Each shifted route is matched with an in-distribution counterpart, isolating the effect of the shift and turning qualitative failures into quantitative diagnostics. Evaluating multiple state-of-the-art models reveals consistent degradation, with an average success-rate drop of 22.8\%. Our analysis uncovers unexpected failure modes, such as ignoring objects clearly visible in the LiDAR and failing to learn the fundamental concepts of free and occupied space. To accelerate follow-up work, Fail2Drive includes an open-source toolbox for creating new scenarios and validating solvability via a privileged expert policy. Together, these components establish a reproducible foundation for benchmarking and improving closed-loop driving generalization. We open-source all code, data, and tools at https://github.com/autonomousvision/fail2drive .

翻译：分布偏移下的泛化能力仍然是封闭式自动驾驶的核心瓶颈。尽管CARLA等仿真器支持安全可扩展的测试，但现有基准测试很少衡量真正的泛化能力：它们通常在测试时重复使用训练场景。因此，成功可能反映的是记忆而非鲁棒驾驶行为。我们提出Fail2Drive，这是首个用于CARLA中封闭式泛化能力的配对路线基准测试，包含200条路线和17种新场景类别，涵盖外观、布局、行为模式和鲁棒性偏移。每条偏移路线均与一个分布内对应路线配对，从而隔离偏移的影响，并将定性失败转化为定量诊断。评估多个最先进模型揭示了持续的性能退化，平均成功率下降22.8%。我们的分析发现了意想不到的失败模式，例如忽略激光雷达中清晰可见的物体，以及未能学习自由空间和占用空间的基本概念。为了加速后续研究，Fail2Drive包含一个开源工具箱，用于创建新场景并通过特权专家策略验证可解性。这些组件共同为封闭式驾驶泛化能力的基准测试和改进建立了可复现的基础。我们在https://github.com/autonomousvision/fail2drive 开源所有代码、数据和工具。