Current scene graph datasets suffer from strong long-tail distributions of their predicate classes. Due to a very low number of some predicate classes in the test sets, no reliable metrics can be retrieved for the rarest classes. We construct a new panoptic scene graph dataset and a set of metrics that are designed as a benchmark for the predictive performance especially on rare predicate classes. To construct the new dataset, we propose a model-assisted annotation pipeline that efficiently finds rare predicate classes that are hidden in a large set of images like needles in a haystack. Contrary to prior scene graph datasets, Haystack contains explicit negative annotations, i.e. annotations that a given relation does not have a certain predicate class. Negative annotations are helpful especially in the field of scene graph generation and open up a whole new set of possibilities to improve current scene graph generation models. Haystack is 100% compatible with existing panoptic scene graph datasets and can easily be integrated with existing evaluation pipelines. Our dataset and code can be found here: https://lorjul.github.io/haystack/. It includes annotation files and simple to use scripts and utilities, to help with integrating our dataset in existing work.

翻译：当前场景图数据集在其谓语类别上普遍存在严重的长尾分布问题。由于测试集中某些谓语类别的数量极少，我们无法为最罕见的类别获取可靠的评估指标。本文构建了一个新的全景场景图数据集及一组评估指标，旨在为模型预测性能（尤其是针对罕见谓语类别）提供基准测试。为构建该数据集，我们提出了一种模型辅助标注流程，能够高效发现隐藏于海量图像中的罕见谓语类别（如同从干草堆中寻针）。与先前的场景图数据集不同，Haystack包含显式的负标注——即标注出某个给定关系不具有特定谓语类别的实例。负标注在场景图生成领域尤为有用，为改进现有场景图生成模型开辟了全新可能性。Haystack与现有全景场景图数据集100%兼容，可轻松集成到现有评估流程中。我们的数据集与代码可通过以下链接获取：https://lorjul.github.io/haystack/，其中包含标注文件及简单易用的脚本与工具，便于将本数据集集成至现有研究工作中。