Handling sparse and unstructured geometric data, such as point clouds or event-based vision, is a pressing challenge in the field of machine vision. Recently, sequence models such as Transformers and state-space models entered the domain of geometric data. These methods require specialized preprocessing to create a sequential view of a set of points. Furthermore, prior works involving sequence models iterate geometric data with either uniform or learned step sizes, implicitly relying on the model to infer the underlying geometric structure. In this work, we propose to encode geometric structure explicitly into the parameterization of a state-space model. State-space models are based on linear dynamics governed by a one-dimensional variable such as time or a spatial coordinate. We exploit this dynamic variable to inject relative differences of coordinates into the step size of the state-space model. The resulting geometric operation computes interactions between all pairs of N points in O(N) steps. Our model deploys the Mamba selective state-space model with a modified CUDA kernel to efficiently map sparse geometric data to modern hardware. The resulting sequence model, which we call STREAM, achieves competitive results on a range of benchmarks from point-cloud classification to event-based vision and audio classification. STREAM demonstrates a powerful inductive bias for sparse geometric data by improving the PointMamba baseline when trained from scratch on the ModelNet40 and ScanObjectNN point cloud analysis datasets. It further achieves, for the first time, 100% test accuracy on all 11 classes of the DVS128 Gestures dataset.

翻译：处理稀疏且非结构化的几何数据（如点云或基于事件的视觉数据）是机器视觉领域的一项紧迫挑战。最近，诸如Transformer和状态空间模型等序列模型进入了几何数据领域。这些方法需要专门的预处理来创建点集的序列视图。此外，先前涉及序列模型的工作以均匀或学习得到的步长迭代几何数据，这隐式地依赖于模型来推断底层的几何结构。在本工作中，我们提出将几何结构显式地编码到状态空间模型的参数化中。状态空间模型基于由时间或空间坐标等一维变量控制的线性动力学。我们利用这一动态变量，将坐标的相对差异注入到状态空间模型的步长中。由此产生的几何操作以O(N)步数计算所有N个点对之间的相互作用。我们的模型部署了Mamba选择性状态空间模型，并采用修改后的CUDA内核，以高效地将稀疏几何数据映射到现代硬件上。我们将所得的序列模型称为STREAM，该模型在从点云分类到基于事件的视觉和音频分类的一系列基准测试中取得了具有竞争力的结果。通过在ModelNet40和ScanObjectNN点云分析数据集上从头开始训练，STREAM相较于PointMamba基线模型表现出改进，这证明了其对稀疏几何数据具有强大的归纳偏置。此外，它首次在DVS128手势数据集的所有11个类别上实现了100%的测试准确率。