This paper presents a novel cross-modal visuo-tactile perception framework for the 3D shape reconstruction of deformable linear objects (DLOs), with a specific focus on cables subject to severe visual occlusions. Unlike existing methods relying predominantly on vision, whose performance degrades under varying illumination, background clutter, or partial visibility, the proposed approach integrates foundation-model-based visual perception with adaptive tactile exploration. The visual pipeline exploits SAM for instance segmentation and Florence for semantic refinement, followed by skeletonization, endpoint detection, and point-cloud extraction. Occluded cable segments are autonomously identified and explored with a tactile sensor, which provides local point clouds that are merged with the visual data through Euclidean clustering and topology-preserving fusion. A B-spline interpolation driven by endpoint-guided point sorting yields a smooth and complete reconstruction of the cable shape. Experimental validation using a robotic manipulator equipped with an RGB-D camera and a tactile pad demonstrates that the proposed framework accurately reconstructs both simple and highly curved single or multiple cable configurations, even when large portions are occluded. These results highlight the potential of foundation-model-enhanced cross-modal perception for advancing robotic manipulation of deformable objects.

翻译：本文提出了一种新颖的跨模态视觉-触觉感知框架，用于可变形线性物体（DLOs）的三维形状重建，特别关注存在严重视觉遮挡的线缆。与主要依赖视觉的现有方法（其性能在光照变化、背景杂乱或局部可见性差的情况下会下降）不同，所提出的方法将基于基础模型的视觉感知与自适应触觉探索相结合。视觉流程利用SAM进行实例分割，利用Florence进行语义细化，随后进行骨架化、端点检测和点云提取。被遮挡的线缆段被自主识别并通过触觉传感器进行探索，该传感器提供局部点云，这些点云通过欧几里得聚类和拓扑保持融合与视觉数据合并。由端点引导的点排序驱动的B样条插值，最终生成线缆形状的平滑完整重建。使用配备RGB-D相机和触觉垫的机器人操纵器进行的实验验证表明，即使存在大范围遮挡，所提出的框架也能准确重建简单和高度弯曲的单根或多根线缆构型。这些结果突显了基础模型增强的跨模态感知在推进机器人对可变形物体操作方面的潜力。