Recent State Space Models (SSMs) such as S4, S5, and Mamba have shown remarkable computational benefits in long-range temporal dependency modeling. However, in many sequence modeling problems, the underlying process is inherently modular and it is of interest to have inductive biases that mimic this modular structure. In this paper, we introduce SlotSSMs, a novel framework for incorporating independent mechanisms into SSMs to preserve or encourage separation of information. Unlike conventional SSMs that maintain a monolithic state vector, SlotSSMs maintains the state as a collection of multiple vectors called slots. Crucially, the state transitions are performed independently per slot with sparse interactions across slots implemented via the bottleneck of self-attention. In experiments, we evaluate our model in object-centric video understanding, 3D visual reasoning, and video prediction tasks, which involve modeling multiple objects and their long-range temporal dependencies. We find that our proposed design offers substantial performance gains over existing sequence modeling methods.

翻译：近年来，状态空间模型（SSMs）如S4、S5和Mamba在长程时间依赖性建模中展现出显著的计算优势。然而，在许多序列建模问题中，底层过程本质上是模块化的，因此引入能够模拟这种模块化结构的归纳偏置具有重要意义。本文提出SlotSSMs，这是一种将独立机制融入SSMs以保持或促进信息分离的新框架。与传统SSMs维护单一整体状态向量不同，SlotSSMs将状态维护为多个称为“槽位”的向量集合。关键在于，状态转移在每个槽位上独立执行，槽位间通过自注意力机制的瓶颈实现稀疏交互。在实验中，我们在以对象为中心的视频理解、三维视觉推理和视频预测任务中评估了模型性能，这些任务涉及对多个对象及其长程时间依赖关系的建模。结果表明，我们提出的设计相较于现有序列建模方法能带来显著的性能提升。