The task of dynamic scene graph generation (SGG) from videos is complicated and challenging due to the inherent dynamics of a scene, temporal fluctuation of model predictions, and the long-tailed distribution of the visual relationships in addition to the already existing challenges in image-based SGG. Existing methods for dynamic SGG have primarily focused on capturing spatio-temporal context using complex architectures without addressing the challenges mentioned above, especially the long-tailed distribution of relationships. This often leads to the generation of biased scene graphs. To address these challenges, we introduce a new framework called TEMPURA: TEmporal consistency and Memory Prototype guided UnceRtainty Attenuation for unbiased dynamic SGG. TEMPURA employs object-level temporal consistencies via transformer-based sequence modeling, learns to synthesize unbiased relationship representations using memory-guided training, and attenuates the predictive uncertainty of visual relations using a Gaussian Mixture Model (GMM). Extensive experiments demonstrate that our method achieves significant (up to 10% in some cases) performance gain over existing methods highlighting its superiority in generating more unbiased scene graphs.

翻译：动态场景图生成（SGG）任务由于场景固有的动态性、模型预测的时间波动性以及视觉关系的长尾分布（除图像级SGG已有的挑战外）而变得复杂且具有挑战性。现有的动态SGG方法主要聚焦于利用复杂架构捕捉时空上下文信息，但并未解决上述挑战（尤其是关系的长尾分布），这往往导致生成有偏的场景图。为解决这些问题，我们提出名为TEMPURA（TEmporal consistency and Memory Prototype guided UnceRtainty Attenuation）的新框架，用于无偏动态SGG。TEMPURA通过基于Transformer的序列建模实现对象级时间一致性，利用记忆引导训练学习合成无偏关系表征，并采用高斯混合模型（GMM）衰减视觉关系的预测不确定性。大量实验表明，我们的方法在性能上显著优于现有方法（部分场景提升达10%），凸显了其在生成更无偏场景图方面的优越性。