We present HetroD, a dataset and benchmark for developing autonomous driving systems in heterogeneous environments. HetroD targets the critical challenge of navi- gating real-world heterogeneous traffic dominated by vulner- able road users (VRUs), including pedestrians, cyclists, and motorcyclists that interact with vehicles. These mixed agent types exhibit complex behaviors such as hook turns, lane splitting, and informal right-of-way negotiation. Such behaviors pose significant challenges for autonomous vehicles but remain underrepresented in existing datasets focused on structured, lane-disciplined traffic. To bridge the gap, we collect a large- scale drone-based dataset to provide a holistic observation of traffic scenes with centimeter-accurate annotations, HD maps, and traffic signal states. We further develop a modular toolkit for extracting per-agent scenarios to support downstream task development. In total, the dataset comprises over 65.4k high- fidelity agent trajectories, 70% of which are from VRUs. HetroD supports modeling of VRU behaviors in dense, het- erogeneous traffic and provides standardized benchmarks for forecasting, planning, and simulation tasks. Evaluation results reveal that state-of-the-art prediction and planning models struggle with the challenges presented by our dataset: they fail to predict lateral VRU movements, cannot handle unstructured maneuvers, and exhibit limited performance in dense and multi-agent scenarios, highlighting the need for more robust approaches to heterogeneous traffic. See our project page for more examples: https://hetroddata.github.io/HetroD/

翻译：本文提出HetroD，一个用于异构环境下自动驾驶系统开发的数据集与基准。HetroD针对以弱势道路使用者（VRU）为主导的真实世界异构交通导航这一关键挑战，包括与车辆交互的行人、骑自行车者和摩托车骑手。这些混合智能体类型表现出复杂行为，如钩形转弯、车道分割和非正式路权协商。此类行为对自动驾驶车辆构成重大挑战，但在现有专注于结构化、车道规范化交通的数据集中仍未被充分表征。为弥补这一差距，我们采集了一个大规模基于无人机的数据集，以厘米级精度标注、高清地图和交通信号状态提供交通场景的整体观测。我们进一步开发了一个模块化工具包，用于提取单智能体场景以支持下游任务开发。该数据集总计包含超过6.54万条高保真智能体轨迹，其中70%来自弱势道路使用者。HetroD支持密集异构交通中弱势道路使用者行为的建模，并为预测、规划与仿真任务提供标准化基准。评估结果表明，当前最先进的预测与规划模型难以应对本数据集提出的挑战：它们无法预测弱势道路使用者的横向移动，不能处理非结构化机动行为，且在密集多智能体场景中表现有限，这凸显了异构交通领域需要更鲁棒的方法。更多示例请访问项目页面：https://hetroddata.github.io/HetroD/