Recently, sparse 3D convolutions have changed 3D object detection. Performing on par with the voting-based approaches, 3D CNNs are memory-efficient and scale to large scenes better. However, there is still room for improvement. With a conscious, practice-oriented approach to problem-solving, we analyze the performance of such methods and localize the weaknesses. Applying modifications that resolve the found issues one by one, we end up with TR3D: a fast fully-convolutional 3D object detection model trained end-to-end, that achieves state-of-the-art results on the standard benchmarks, ScanNet v2, SUN RGB-D, and S3DIS. Moreover, to take advantage of both point cloud and RGB inputs, we introduce an early fusion of 2D and 3D features. We employ our fusion module to make conventional 3D object detection methods multimodal and demonstrate an impressive boost in performance. Our model with early feature fusion, which we refer to as TR3D+FF, outperforms existing 3D object detection approaches on the SUN RGB-D dataset. Overall, besides being accurate, both TR3D and TR3D+FF models are lightweight, memory-efficient, and fast, thereby marking another milestone on the way toward real-time 3D object detection. Code is available at https://github.com/SamsungLabs/tr3d .

翻译：最近，稀疏三维卷积改变了三维目标检测。与基于投票的方法性能相当，三维CNN内存高效且能更好地扩展到大场景。然而，仍有改进空间。通过有意识、面向实践的问题解决方法，我们分析了此类方法的性能并定位了弱点。通过逐一应用解决已发现问题的改进措施，我们最终得到TR3D：一个端到端训练的快速全卷积三维目标检测模型，在标准基准ScanNet v2、SUN RGB-D和S3DIS上取得领先结果。此外，为同时利用点云和RGB输入，我们引入了一种二维与三维特征的早期融合方法。我们利用该融合模块使传统三维目标检测方法转为多模态，并展现出显著的性能提升。融入早期特征融合的模型（称为TR3D+FF）在SUN RGB-D数据集上优于现有三维目标检测方法。总体而言，TR3D和TR3D+FF模型不仅准确，而且轻量、内存高效且快速，从而在通往实时三维目标检测的道路上树立了又一个里程碑。代码开源地址：https://github.com/SamsungLabs/tr3d。