Classification of different object surface material types can play a significant role in the decision-making algorithms for mobile robots and autonomous vehicles. RGB-based scene-level semantic segmentation has been well-addressed in the literature. However, improving material recognition using the depth modality and its integration with SLAM algorithms for 3D semantic mapping could unlock new potential benefits in the robotics perception pipeline. To this end, we propose a complementarity-aware deep learning approach for RGB-D-based material classification built on top of an object-oriented pipeline. The approach further integrates the ORB-SLAM2 method for 3D scene mapping with multiscale clustering of the detected material semantics in the point cloud map generated by the visual SLAM algorithm. Extensive experimental results with existing public datasets and newly contributed real-world robot datasets demonstrate a significant improvement in material classification and 3D clustering accuracy compared to state-of-the-art approaches for 3D semantic scene mapping.

翻译：不同物体表面材料类型的分类在移动机器人与自动驾驶车辆的决策算法中可发挥重要作用。基于RGB的场景级语义分割已在文献中得到充分研究。然而，利用深度模态改进材料识别，并将其与SLAM算法结合以实现三维语义建图，有望为机器人感知流程带来新的潜在优势。为此，我们提出一种基于互补性感知的深度学习方法，用于RGB-D材料分类，该方法构建于面向对象的处理流程之上。该方案进一步将ORB-SLAM2三维场景建图方法与视觉SLAM算法生成的点云图中检测到的材料语义多尺度聚类相结合。通过对现有公开数据集及新贡献的真实世界机器人数据集进行大量实验，结果表明：相较于三维语义场景建图领域的最先进方法，本方法在材料分类与三维聚类精度方面均取得显著提升。