This paper presents a novel vision-based proprioception approach for a soft robotic finger capable of estimating and reconstructing tactile interactions in terrestrial and aquatic environments. The key to this system lies in the finger's unique metamaterial structure, which facilitates omni-directional passive adaptation during grasping, protecting delicate objects across diverse scenarios. A compact in-finger camera captures high-framerate images of the finger's deformation during contact, extracting crucial tactile data in real time. We present a method of the volumetric discretized model of the soft finger and use the geometry constraints captured by the camera to find the optimal estimation of the deformed shape. The approach is benchmarked with a motion-tracking system with sparse markers and a haptic device with dense measurements. Both results show state-of-the-art accuracies, with a median error of 1.96 mm for overall body deformation, corresponding to 2.1$\%$ of the finger's length. More importantly, the state estimation is robust in both on-land and underwater environments as we demonstrate its usage for underwater object shape sensing. This combination of passive adaptation and real-time tactile sensing paves the way for amphibious robotic grasping applications.

翻译：本文提出了一种新颖的基于视觉的本体感知方法，应用于能够在陆地和水下环境中估计与重建触觉交互的软体机器人手指。该系统的关键在于手指独特的超材料结构，该结构在抓取过程中实现全方位被动适应，从而在多种场景下保护易碎物体。紧凑的手指内摄像头以高帧率捕捉接触过程中手指的形变图像，实时提取关键触觉数据。我们提出了一种软体手指的体积离散化模型方法，并利用摄像头捕捉的几何约束来获取形变形状的最优估计。该方法与采用稀疏标记点的运动追踪系统以及采用密集测量的触觉设备进行了基准测试。两种结果均显示出最先进的精度，整体身体形变的中位误差为 1.96 毫米，相当于手指长度的 2.1%。更重要的是，该状态估计在陆地和水中环境中均表现出鲁棒性，我们通过其在水下物体形状感知中的应用进行了验证。这种被动适应与实时触觉感知的结合为两栖机器人抓取应用铺平了道路。