Video scene graph generation (VidSGG) aims to identify objects in visual scenes and infer their relationships for a given video. It requires not only a comprehensive understanding of each object scattered on the whole scene but also a deep dive into their temporal motions and interactions. Inherently, object pairs and their relationships enjoy spatial co-occurrence correlations within each image and temporal consistency/transition correlations across different images, which can serve as prior knowledge to facilitate VidSGG model learning and inference. In this work, we propose a spatial-temporal knowledge-embedded transformer (STKET) that incorporates the prior spatial-temporal knowledge into the multi-head cross-attention mechanism to learn more representative relationship representations. Specifically, we first learn spatial co-occurrence and temporal transition correlations in a statistical manner. Then, we design spatial and temporal knowledge-embedded layers that introduce the multi-head cross-attention mechanism to fully explore the interaction between visual representation and the knowledge to generate spatial- and temporal-embedded representations, respectively. Finally, we aggregate these representations for each subject-object pair to predict the final semantic labels and their relationships. Extensive experiments show that STKET outperforms current competing algorithms by a large margin, e.g., improving the mR@50 by 8.1%, 4.7%, and 2.1% on different settings over current algorithms.

翻译：视频场景图生成（VidSGG）旨在识别视觉场景中的物体并推断它们在给定视频中的关系。这不仅需要全面理解分散在整个场景中的每个物体，还需要深入探索它们的时间运动与交互。本质上，物体对及其关系在每张图像内具有空间共现相关性，在不同图像间具有时间一致性/过渡相关性，这些可作为先验知识促进VidSGG模型的学习与推理。本研究提出一种时空知识嵌入Transformer（STKET），将先验时空知识融入多头交叉注意力机制中，以学习更具代表性的关系表示。具体而言，我们首先以统计方式学习空间共现和时间过渡相关性；随后设计空间与时间知识嵌入层，利用多头交叉注意力机制充分探索视觉表示与知识之间的交互，分别生成空间嵌入表示和时间嵌入表示；最后为每个主客体对聚合这些表示，以预测最终的语义标签及其关系。大量实验表明，STKET以显著优势超越现有竞争算法——例如在不同设置下mR@50指标分别提升8.1%、4.7%和2.1%。