The current approach for testing the robustness of object detectors suffers from serious deficiencies such as improper methods of performing out-of-distribution detection and using calibration metrics which do not consider both localisation and classification quality. In this work, we address these issues, and introduce the Self-Aware Object Detection (SAOD) task, a unified testing framework which respects and adheres to the challenges that object detectors face in safety-critical environments such as autonomous driving. Specifically, the SAOD task requires an object detector to be: robust to domain shift; obtain reliable uncertainty estimates for the entire scene; and provide calibrated confidence scores for the detections. We extensively use our framework, which introduces novel metrics and large scale test datasets, to test numerous object detectors in two different use-cases, allowing us to highlight critical insights into their robustness performance. Finally, we introduce a simple baseline for the SAOD task, enabling researchers to benchmark future proposed methods and move towards robust object detectors which are fit for purpose. Code is available at https://github.com/fiveai/saod

翻译：当前针对目标检测器鲁棒性的测试方法存在严重缺陷，例如不恰当的非分布外检测方法，以及使用未同时考虑定位与分类质量的校准指标。本研究针对这些问题，提出了自我感知目标检测任务（Self-Aware Object Detection, SAOD），这是一个统一测试框架，旨在应对目标检测器在自动驾驶等安全关键环境中面临的挑战。具体而言，SAOD任务要求目标检测器具备以下能力：对域偏移具有鲁棒性；为整个场景获得可靠的不确定性估计；为检测结果提供校准后的置信度分数。我们广泛运用该框架（其中引入了新颖的评估指标和大规模测试数据集），针对两个不同应用场景测试了多种目标检测器，从而揭示了其鲁棒性能的关键见解。最后，我们为SAOD任务提供了一个简单基线，使研究人员能够对未来的方法进行基准测试，推动构建适用于实际场景的鲁棒目标检测器。代码已开源至 https://github.com/fiveai/saod