Research in scene graph generation (SGG) usually considers two-stage models, that is, detecting a set of entities, followed by combining them and labeling all possible relationships. While showing promising results, the pipeline structure induces large parameter and computation overhead, and typically hinders end-to-end optimizations. To address this, recent research attempts to train single-stage models that are computationally efficient. With the advent of DETR, a set based detection model, one-stage models attempt to predict a set of subject-predicate-object triplets directly in a single shot. However, SGG is inherently a multi-task learning problem that requires modeling entity and predicate distributions simultaneously. In this paper, we propose Transformers with conditional queries for SGG, namely, TraCQ with a new formulation for SGG that avoids the multi-task learning problem and the combinatorial entity pair distribution. We employ a DETR-based encoder-decoder design and leverage conditional queries to significantly reduce the entity label space as well, which leads to 20% fewer parameters compared to state-of-the-art single-stage models. Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats many state-of-the-art two-stage methods on the Visual Genome dataset, yet is capable of end-to-end training and faster inference.

翻译：场景图生成（SGG）的研究通常采用两阶段模型，即先检测实体集合，再组合这些实体并标注所有可能的关系。尽管该方法取得了显著成果，但其流水线结构导致参数和计算开销过大，通常难以实现端到端优化。为此，近期研究尝试训练计算高效的单阶段模型。随着基于集合检测模型DETR的出现，单阶段模型试图通过一次前向预测直接输出一组"主体-谓词-客体"三元组。然而，SGG本质上是一个多任务学习问题，需要同时建模实体分布和谓词分布。本文提出基于条件查询的SGG变换器（TraCQ），通过新型SGG公式规避了多任务学习问题与组合式实体对分布。我们采用基于DETR的编码器-解码器架构，并利用条件查询大幅减少实体标签空间，相比最先进的单阶段模型参数减少20%。实验结果表明，TraCQ不仅优于现有单阶段场景图生成方法，在Visual Genome数据集上还超越了许多最先进的两阶段方法，同时支持端到端训练与更快的推理速度。