Scene graph generation (SGG) endeavors to predict visual relationships between pairs of objects within an image. Prevailing SGG methods traditionally assume a one-off learning process for SGG. This conventional paradigm may necessitate repetitive training on all previously observed samples whenever new relationships emerge, mitigating the risk of forgetting previously acquired knowledge. This work seeks to address this pitfall inherent in a suite of prior relationship predictions. Motivated by the achievements of in-context learning in pretrained language models, our approach imbues the model with the capability to predict relationships and continuously acquire novel knowledge without succumbing to catastrophic forgetting. To achieve this goal, we introduce a novel and pragmatic framework for scene graph generation, namely Lifelong Scene Graph Generation (LSGG), where tasks, such as predicates, unfold in a streaming fashion. In this framework, the model is constrained to exclusive training on the present task, devoid of access to previously encountered training data, except for a limited number of exemplars, but the model is tasked with inferring all predicates it has encountered thus far. Rigorous experiments demonstrate the superiority of our proposed method over state-of-the-art SGG models in the context of LSGG across a diverse array of metrics. Besides, extensive experiments on the two mainstream benchmark datasets, VG and Open-Image(v6), show the superiority of our proposed model to a number of competitive SGG models in terms of continuous learning and conventional settings. Moreover, comprehensive ablation experiments demonstrate the effectiveness of each component in our model.

翻译：场景图生成（SGG）旨在预测图像中成对物体之间的视觉关系。现有的SGG方法通常假设一次性学习过程。然而，这种传统范式在遇到新关系时，可能需要对所有先前观察到的样本进行重复训练，以缓解遗忘已有知识的风险。本文旨在解决先前关系预测中普遍存在的这一缺陷。受预训练语言模型中上下文学习成功经验的启发，我们的方法赋予模型预测关系并持续获取新知识的能力，同时避免灾难性遗忘。为此，我们提出了一种新颖且实用的场景图生成框架——终身场景图生成（LSGG），其中任务（如谓词）以流式方式展开。在该框架中，模型仅针对当前任务进行训练，除少量示例外无法访问先前遇到的训练数据，但需对所有已见过的谓词进行推理。大量实验表明，在LSGG场景下，我们的方法在多种指标上均优于最先进的SGG模型。此外，在两个主流基准数据集VG和Open-Image(v6)上的广泛实验显示，无论是在持续学习还是传统设置中，我们的模型均优于多个具有竞争力的SGG模型。同时，全面的消融实验证明了模型各组件的有效性。