Spatio-temporal scene graphs provide a principled representation for modeling evolving object interactions, yet existing methods remain fundamentally frame-centric: they reason only about currently visible objects, discard entities upon occlusion, and operate in 2D. To address this, we first introduce ActionGenome4D, a dataset that upgrades Action Genome videos into 4D scenes via feed-forward 3D reconstruction, world-frame oriented bounding boxes for every object involved in actions, and dense relationship annotations including for objects that are temporarily unobserved due to occlusion or camera motion. Building on this data, we formalize World Scene Graph Generation (WSGG), the task of constructing a world scene graph at each timestamp that encompasses all interacting objects in the scene, both observed and unobserved. We then propose three complementary methods, each exploring a different inductive bias for reasoning about unobserved objects: PWG (Persistent World Graph), which implements object permanence via a zero-order feature buffer; MWAE (Masked World Auto-Encoder), which reframes unobserved-object reasoning as masked completion with cross-view associative retrieval; and 4DST (4D Scene Transformer), which replaces the static buffer with differentiable per-object temporal attention enriched by 3D motion and camera-pose features. We further design and evaluate the performance of strong open-source Vision-Language Models on the WSGG task via a suite of Graph RAG-based approaches, establishing baselines for unlocalized relationship prediction. WSGG thus advances video scene understanding toward world-centric, temporally persistent, and interpretable scene reasoning.

翻译：时空场景图为建模演化中的物体交互提供了规范化表示，然而现有方法本质上仍局限于帧层面：仅对当前可见物体进行推理，在遮挡时丢弃实体，且仅在二维空间操作。为解决此问题，我们首先提出ActionGenome4D数据集，该数据集通过前馈式三维重建将Action Genome视频升级为四维场景，为动作涉及的每个物体提供世界坐标系下的定向边界框，并包含密集关系标注（包括因遮挡或相机运动而暂时未被观测到的物体）。基于此数据，我们形式化定义了世界场景图生成任务，即在每个时间戳构建包含场景中所有交互物体（包括已观测和未观测物体）的世界场景图。随后我们提出三种互补方法，分别探索不同的归纳偏置以推理未观测物体：PWG（持久世界图）通过零阶特征缓冲区实现物体恒存性；MWAE（掩码世界自编码器）将未观测物体推理重构为基于跨视图关联检索的掩码补全任务；4DST（四维场景变换器）则用可微分逐物体时序注意力机制替代静态缓冲区，并融入三维运动与相机位姿特征进行增强。我们进一步通过一套基于图检索增强生成的方法，设计并评估了开源视觉语言模型在WSGG任务上的性能，为无定位关系预测建立了基线。WSGG由此将视频场景理解推向以世界为中心、具有时序持续性且可解释的场景推理新阶段。