Panoptic Scene Graph (PSG) is a challenging task in Scene Graph Generation (SGG) that aims to create a more comprehensive scene graph representation using panoptic segmentation instead of boxes. Compared to SGG, PSG has several challenging problems: pixel-level segment outputs and full relationship exploration (It also considers thing and stuff relation). Thus, current PSG methods have limited performance, which hinders downstream tasks or applications. The goal of this work aims to design a novel and strong baseline for PSG. To achieve that, we first conduct an in-depth analysis to identify the bottleneck of the current PSG models, finding that inter-object pair-wise recall is a crucial factor that was ignored by previous PSG methods. Based on this and the recent query-based frameworks, we present a novel framework: Pair then Relation (Pair-Net), which uses a Pair Proposal Network (PPN) to learn and filter sparse pair-wise relationships between subjects and objects. Moreover, we also observed the sparse nature of object pairs for both Motivated by this, we design a lightweight Matrix Learner within the PPN, which directly learn pair-wised relationships for pair proposal generation. Through extensive ablation and analysis, our approach significantly improves upon leveraging the segmenter solid baseline. Notably, our method achieves new state-of-the-art results on the PSG benchmark, with over 10\% absolute gains compared to PSGFormer. The code of this paper is publicly available at https://github.com/king159/Pair-Net.

翻译：全景场景图（PSG）是场景图生成（SGG）中的一项具有挑战性的任务，旨在利用全景分割而非边界框来创建更全面的场景图表示。与SGG相比，PSG面临若干难题：像素级分割输出和完整关系探索（它还考虑物体与材质关系）。因此，当前PSG方法的性能有限，这阻碍了下游任务或应用的发展。本工作的目标是为PSG设计一种新颖且强力的基线方法。为实现这一点，我们首先进行深入分析以识别当前PSG模型的瓶颈，发现物体间成对召回率是先前PSG方法忽略的关键因素。基于此和最近的查询式框架，我们提出一种新颖框架：先配对再关系（Pair-Net），它使用配对提议网络（PPN）来学习和过滤主客体之间的稀疏成对关系。此外，我们还观察到物体对的稀疏性。受此启发，我们在PPN内设计了一个轻量级矩阵学习器（Matrix Learner），它直接学习成对关系以生成配对提议。通过广泛的消融和分析，我们的方法显著提升了基于分割器的坚实基线性能。值得注意的是，我们的方法在PSG基准测试上达到了新的最优结果，相比PSGFormer具有超过10%的绝对收益。本文代码已在https://github.com/king159/Pair-Net公开。