We introduce PointConvFormer, a novel building block for point cloud based deep network architectures. Inspired by generalization theory, PointConvFormer combines ideas from point convolution, where filter weights are only based on relative position, and Transformers which utilize feature-based attention. In PointConvFormer, attention computed from feature difference between points in the neighborhood is used to modify the convolutional weights at each point. Hence, we preserved the invariances from point convolution, whereas attention helps to select relevant points in the neighborhood for convolution. PointConvFormer is suitable for multiple tasks that require details at the point level, such as segmentation and scene flow estimation tasks. We experiment on both tasks with multiple datasets including ScanNet, SemanticKitti, FlyingThings3D and KITTI. Our results show that PointConvFormer offers a better accuracy-speed tradeoff than classic convolutions, regular transformers, and voxelized sparse convolution approaches. Visualizations show that PointConvFormer performs similarly to convolution on flat areas, whereas the neighborhood selection effect is stronger on object boundaries, showing that it has got the best of both worlds.

翻译：我们提出PointConvFormer，这是一种用于基于点云的深度网络架构的新型构建模块。受泛化理论启发，PointConvFormer结合了点卷积（其中滤波器权重仅基于相对位置）和Transformer（利用基于特征的注意力机制）的思想。在PointConvFormer中，基于邻域内点之间特征差异计算的注意力被用于修改每个点的卷积权重。因此，我们保留了点卷积的不变性，而注意力则有助于选择邻域中相关的点进行卷积。PointConvFormer适用于需要点级细节的多种任务，例如分割和场景流估计任务。我们在多个数据集上对这两项任务进行了实验，包括ScanNet、SemanticKitti、FlyingThings3D和KITTI。我们的结果表明，与经典卷积、常规Transformer和体素化稀疏卷积方法相比，PointConvFormer提供了更好的精度-速度权衡。可视化显示，PointConvFormer在平坦区域上的表现类似于卷积，而在物体边界上的邻域选择效果更强，表明它兼具了两者的优势。