Before deployment in the real-world deep neural networks require thorough evaluation of how they handle both knowns, inputs represented in the training data, and unknowns (anomalies). This is especially important for scene understanding tasks with safety critical applications, such as in autonomous driving. Existing datasets allow evaluation of only knowns or unknowns - but not both, which is required to establish "in the wild" suitability of deep neural network models. To bridge this gap, we propose a novel anomaly segmentation dataset, ISSU, that features a diverse set of anomaly inputs from cluttered real-world environments. The dataset is twice larger than existing anomaly segmentation datasets, and provides a training, validation and test set for controlled in-domain evaluation. The test set consists of a static and temporal part, with the latter comprised of videos. The dataset provides annotations for both closed-set (knowns) and anomalies, enabling closed-set and open-set evaluation. The dataset covers diverse conditions, such as domain and cross-sensor shift, illumination variation and allows ablation of anomaly detection methods with respect to these variations. Evaluation results of current state-of-the-art methods confirm the need for improvements especially in domain-generalization, small and large object segmentation.

翻译：在现实世界部署之前，深度神经网络需要对其如何处理训练数据中已表征的已知输入与未知输入（异常）进行全面评估。这对于具有安全关键应用场景的场景理解任务（如自动驾驶）尤为重要。现有数据集仅支持对已知或未知输入进行评估，而无法同时评估两者——而这正是确立深度神经网络模型"在野外"适用性所必需的。为弥补这一差距，我们提出了新型异常分割数据集ISSU，其特点是包含来自杂乱现实环境中的多样化异常输入。该数据集规模是现有异常分割数据集的两倍，并提供了用于受控域内评估的训练集、验证集和测试集。测试集包含静态和时序两部分，后者由视频序列构成。数据集同时提供闭集（已知类别）与异常类别的标注，支持闭集与开集评估。数据集涵盖多样化条件，如域偏移与跨传感器偏移、光照变化等，并支持针对这些变量对异常检测方法进行消融研究。当前最先进方法的评估结果证实了改进的必要性，特别是在域泛化能力、小目标与大目标分割方面。