Scene Graph Generation (SGG) is a challenging task of detecting objects and predicting relationships between objects. After DETR was developed, one-stage SGG models based on a one-stage object detector have been actively studied. However, complex modeling is used to predict the relationship between objects, and the inherent relationship between object queries learned in the multi-head self-attention of the object detector has been neglected. We propose a lightweight one-stage SGG model that extracts the relation graph from the various relationships learned in the multi-head self-attention layers of the DETR decoder. By fully utilizing the self-attention by-products, the relation graph can be extracted effectively with a shallow relation extraction head. Considering the dependency of the relation extraction task on the object detection task, we propose a novel relation smoothing technique that adjusts the relation label adaptively according to the quality of the detected objects. By the relation smoothing, the model is trained according to the continuous curriculum that focuses on object detection task at the beginning of training and performs multi-task learning as the object detection performance gradually improves. Furthermore, we propose a connectivity prediction task that predicts whether a relation exists between object pairs as an auxiliary task of the relation extraction. We demonstrate the effectiveness and efficiency of our method for the Visual Genome and Open Image V6 datasets. Our code is publicly available at https://github.com/naver-ai/egtr.

翻译：场景图生成（SGG）是一项检测物体并预测物体间关系的挑战性任务。在DETR提出后，基于单阶段目标检测器的单阶段SGG模型得到了深入研究。然而，现有方法需通过复杂建模来预测物体间关系，但忽视了在目标检测器的多头自注意力机制中学习到的物体查询之间的固有关系。为此，我们提出一种轻量级单阶段SGG模型，该模型能从DETR解码器多头自注意力层中学习到的多种关系里提取关系图。通过充分利用自注意力机制的附带产物，仅需一个浅层关系提取头即可有效提取关系图。考虑到关系提取任务对目标检测任务的依赖性，我们提出一种新颖的关系平滑技术，该技术能根据检测物体的质量自适应调整关系标签。通过关系平滑，模型遵循连续课程进行训练：在训练初期专注于目标检测任务，随着目标检测性能逐步提升，再执行多任务学习。此外，我们提出连通性预测任务，将其作为关系提取的辅助任务，用于预测物体对之间是否存在关系。我们在Visual Genome和Open Image V6数据集上验证了该方法的有效性和高效性。代码已开源至https://github.com/naver-ai/egtr。