Benchmarking is a common method for evaluating trajectory prediction models for autonomous driving. Existing benchmarks rely on datasets, which are biased towards more common scenarios, such as cruising, and distance-based metrics that are computed by averaging over all scenarios. Following such a regiment provides a little insight into the properties of the models both in terms of how well they can handle different scenarios and how admissible and diverse their outputs are. There exist a number of complementary metrics designed to measure the admissibility and diversity of trajectories, however, they suffer from biases, such as length of trajectories. In this paper, we propose a new benChmarking paRadIgm for evaluaTing trajEctoRy predIction Approaches (CRITERIA). Particularly, we propose 1) a method for extracting driving scenarios at varying levels of specificity according to the structure of the roads, models' performance, and data properties for fine-grained ranking of prediction models; 2) A set of new bias-free metrics for measuring diversity, by incorporating the characteristics of a given scenario, and admissibility, by considering the structure of roads and kinematic compliancy, motivated by real-world driving constraints. 3) Using the proposed benchmark, we conduct extensive experimentation on a representative set of the prediction models using the large scale Argoverse dataset. We show that the proposed benchmark can produce a more accurate ranking of the models and serve as a means of characterizing their behavior. We further present ablation studies to highlight contributions of different elements that are used to compute the proposed metrics.

翻译：基准测试是评估自动驾驶轨迹预测模型的常用方法。现有基准测试依赖于数据集，这些数据集偏向于更常见的场景（如巡航），以及基于距离的指标，这些指标通过在所有场景上取平均计算得出。遵循这种模式无法深入洞察模型在不同场景下的处理能力，也无法评估其输出轨迹的合理性与多样性。虽然存在多种用于衡量轨迹合理性与多样性的补充指标，但这些指标存在偏差（如轨迹长度偏差）。本文提出了一种评估轨迹预测方法的新型基准测试范式（CRITERIA）。具体而言，我们提出：1）一种根据道路结构、模型性能和数据属性提取不同粒度驾驶场景的方法，用于实现预测模型的细粒度排名；2）一组新的无偏指标，通过融入特定场景特征来衡量多样性，并基于真实驾驶约束考虑道路结构和运动学兼容性来衡量合理性；3）利用所提出的基准测试框架，在大型Argoverse数据集上对代表性预测模型进行了广泛实验。结果表明，该基准测试能更准确地生成模型排名，并可作为表征模型行为的手段。我们进一步通过消融研究，揭示了计算所提指标时不同要素的贡献。