We propose a novel framework for the task of object-centric video prediction, i.e., extracting the compositional structure of a video sequence, as well as modeling objects dynamics and interactions from visual observations in order to predict the future object states, from which we can then generate subsequent video frames. With the goal of learning meaningful spatio-temporal object representations and accurately forecasting object states, we propose two novel object-centric video predictor (OCVP) transformer modules, which decouple the processing of temporal dynamics and object interactions, thus presenting an improved prediction performance. In our experiments, we show how our object-centric prediction framework utilizing our OCVP predictors outperforms object-agnostic video prediction models on two different datasets, while maintaining consistent and accurate object representations.

翻译：我们提出了一种面向物体视频预测任务的新框架，即提取视频序列的组成结构，同时基于视觉观察建模物体动力学与交互，以预测未来物体状态，并由此生成后续视频帧。为了实现有意义的时空物体表征学习并准确预测物体状态，我们提出了两种新型面向物体视频预测（OCVP）Transformer模块，这些模块将时间动力学处理与物体交互进行解耦，从而提升了预测性能。实验表明，采用我们OCVP预测器的面向物体预测框架在两个不同数据集上均优于与物体无关的视频预测模型，同时保持了稳定且准确的物体表征。