Transformer, as an alternative to CNN, has been proven effective in many modalities (e.g., texts and images). For 3D point cloud transformers, existing efforts focus primarily on pushing their accuracy to the state-of-the-art level. However, their latency lags behind sparse convolution-based models (3x slower), hindering their usage in resource-constrained, latency-sensitive applications (such as autonomous driving). This inefficiency comes from point clouds' sparse and irregular nature, whereas transformers are designed for dense, regular workloads. This paper presents FlatFormer to close this latency gap by trading spatial proximity for better computational regularity. We first flatten the point cloud with window-based sorting and partition points into groups of equal sizes rather than windows of equal shapes. This effectively avoids expensive structuring and padding overheads. We then apply self-attention within groups to extract local features, alternate sorting axis to gather features from different directions, and shift windows to exchange features across groups. FlatFormer delivers state-of-the-art accuracy on Waymo Open Dataset with 4.6x speedup over (transformer-based) SST and 1.4x speedup over (sparse convolutional) CenterPoint. This is the first point cloud transformer that achieves real-time performance on edge GPUs and is faster than sparse convolutional methods while achieving on-par or even superior accuracy on large-scale benchmarks.

翻译：摘要：Transformer作为CNN的替代方案，已在文本、图像等多种模态中被证明有效。针对3D点云Transformer，现有研究主要致力于提升其精度至最优水平。然而，其延迟落后于基于稀疏卷积的模型（慢3倍），阻碍了其在资源受限、对延迟敏感的自主驾驶等场景中的应用。这种低效源于点云稀疏且不规则的本质特性，而Transformer专为密集规则化数据设计。本文提出FlatFormer，通过以空间邻近性换取更好的计算规则性来弥合延迟差距。我们首先通过基于窗口的排序对点云进行展平，并将点划分为大小相等的组（而非形状相等的窗口），从而有效避免昂贵的结构化和填充开销。随后在组内应用自注意力提取局部特征，交替排序轴以从不同方向聚合特征，并采用滑动窗口实现跨组特征交换。在Waymo开放数据集上，FlatFormer实现了最优精度，相比基于Transformer的SST方法提速4.6倍，相比稀疏卷积的CenterPoint方法提速1.4倍。这是首个在边缘GPU上实现实时性能的点云Transformer，其速度超越稀疏卷积方法，同时在大规模基准测试中保持相当甚至更优的精度。