Panoptic Scene Graph generation (PSG) is a recently proposed task in image scene understanding that aims to segment the image and extract triplets of subjects, objects and their relations to build a scene graph. This task is particularly challenging for two reasons. First, it suffers from a long-tail problem in its relation categories, making naive biased methods more inclined to high-frequency relations. Existing unbiased methods tackle the long-tail problem by data/loss rebalancing to favor low-frequency relations. Second, a subject-object pair can have two or more semantically overlapping relations. While existing methods favor one over the other, our proposed HiLo framework lets different network branches specialize on low and high frequency relations, enforce their consistency and fuse the results. To the best of our knowledge we are the first to propose an explicitly unbiased PSG method. In extensive experiments we show that our HiLo framework achieves state-of-the-art results on the PSG task. We also apply our method to the Scene Graph Generation task that predicts boxes instead of masks and see improvements over all baseline methods.

翻译：全景场景图生成（Panoptic Scene Graph Generation, PSG）是近期提出的图像场景理解任务，旨在分割图像并提取主体、客体及其关系的三元组以构建场景图。该任务面临两大挑战：首先，其关系类别存在长尾分布问题，导致朴素的有偏方法更倾向于高频关系，而现有无偏方法通过数据/损失重平衡来偏向低频关系以解决长尾问题；其次，主体-客体对可能包含两个或更多语义重叠的关系。现有方法常倾向于选择其中一种关系，而本文提出的HiLo框架通过让不同网络分支分别擅长低频与高频关系，强制其一致性并融合结果。据我们所知，这是首个显式实现无偏PSG的方法。大量实验表明，HiLo框架在PSG任务上取得了最先进的性能。同时，我们将该方法应用于预测边界框而非掩码的场景图生成任务，在所有基线方法上均获得了性能提升。