Perceiving a three-dimensional (3D) scene with multiple objects while moving indoors is essential for vision-based mobile cobots, especially for enhancing their manipulation tasks. In this work, we present an end-to-end pipeline with instance segmentation, feature matching, and point-set registration for egocentric robots with binocular vision, and demonstrate the robot's grasping capability through the proposed pipeline. First, we design an RGB image-based segmentation approach for single-view 3D semantic scene segmentation, leveraging common object classes in 2D datasets to encapsulate 3D points into point clouds of object instances through corresponding depth maps. Next, 3D correspondences of two consecutive segmented point clouds are extracted based on matched keypoints between objects of interest in RGB images from the prior step. In addition, to be aware of spatial changes in 3D feature distribution, we also weigh each 3D point pair based on the estimated distribution using kernel density estimation (KDE), which subsequently gives robustness with less central correspondences while solving for rigid transformations between point clouds. Finally, we test our proposed pipeline on the 7-DOF dual-arm Baxter robot with a mounted Intel RealSense D435i RGB-D camera. The result shows that our robot can segment objects of interest, register multiple views while moving, and grasp the target object. The source code is available at https://github.com/mkhangg/semantic_scene_perception.

翻译：在室内移动过程中感知包含多个物体的三维场景，对于基于视觉的移动协作机器人而言至关重要，尤其能提升其操作能力。本文提出了一种面向具身双目视觉机器人的端到端流程，融合实例分割、特征匹配与点集配准技术，并通过该流程验证了机器人的抓取能力。首先，我们设计了基于RGB图像的单视角三维语义场景分割方法，利用二维数据集中的通用物体类别，通过对应深度图将三维点封装为物体实例点云。其次，基于前序步骤中RGB图像感兴趣物体间的匹配关键点，提取连续两帧分割点云的三维对应关系。此外，为感知三维特征分布的空间变化，我们采用核密度估计（KDE）对每对三维点进行加权处理，使得求解点云间刚体变换时，非中心对应点对仍能增强算法鲁棒性。最后，我们在搭载Intel RealSense D435i RGB-D相机的七自由度双臂Baxter机器人上测试了所提流程。实验结果表明，机器人能够有效分割感兴趣物体，在移动过程中完成多视角配准，并成功抓取目标物体。相关源代码已开源：https://github.com/mkhangg/semantic_scene_perception。