OctFormer can not only serve as a general and effective backbone for 3D point cloud segmentation and object detection but also have linear complexity and is scalable for large-scale point clouds. The key challenge in applying transformers to point clouds is reducing the quadratic, thus overwhelming, computation complexity of attentions. To combat this issue, several works divide point clouds into non-overlapping windows and constrain attentions in each local window. However, the point number in each window varies greatly, impeding the efficient execution on GPU. Observing that attentions are robust to the shapes of local windows, we propose a novel octree attention, which leverages sorted shuffled keys of octrees to partition point clouds into local windows containing a fixed number of points while permitting shapes of windows to change freely. And we also introduce dilated octree attention to expand the receptive field further. Our octree attention can be implemented in 10 lines of code with open-sourced libraries and runs 17 times faster than other point cloud attentions when the point number exceeds 200k. Built upon the octree attention, OctFormer can be easily scaled up and achieves state-of-the-art performances on a series of 3D segmentation and detection benchmarks, surpassing previous sparse-voxel-based CNNs and point cloud transformers in terms of both efficiency and effectiveness. Notably, on the challenging ScanNet200 dataset, OctFormer outperforms sparse-voxel-based CNNs by 7.3 in mIoU. Our code and trained models are available at https://wang-ps.github.io/octformer.

翻译：OctFormer不仅可作为通用的高效主干网络用于三维点云分割与目标检测，其计算复杂度呈线性且可扩展至大规模点云。将Transformer应用于点云的核心挑战在于降低注意力机制中二次方乃至过高的计算复杂度。为解决该问题，现有工作将点云划分为非重叠窗口并在各局部窗口内约束注意力计算，但每个窗口内点数量差异显著，阻碍了其在GPU上的高效执行。观察到注意力机制对局部窗口形状具有鲁棒性，我们提出新颖的八叉树注意力机制：利用八叉树排序混洗键将点云划分为包含固定点数目的局部窗口，同时允许窗口形状自由变化；并引入膨胀八叉树注意力进一步扩展感受野。该机制仅需10行开源代码即可实现，当点云数量超过20万时，运行速度较其他点云注意力机制提升17倍。基于八叉树注意力构建的OctFormer可轻松扩展，在三维分割与检测系列基准测试中达到最优性能，在效率与效果上均超越此前基于稀疏体素的CNN及点云Transformer。值得关注的是，在极具挑战性的ScanNet200数据集上，OctFormer的mIoU较稀疏体素CNN提升7.3。代码与预训练模型已发布于https://wang-ps.github.io/octformer。