Sensory substitution enables biological systems to perceive stimuli that are typically perceived by another organ, which is inspirational for physical agents. Multimodal perception of intrinsic and extrinsic interactions is critical in building an intelligent robot that learns. This study presents a Vision-based See-Through Perception (VBSeeThruP) architecture that simultaneously perceives multiple intrinsic and extrinsic modalities from a single visual input, in a markerless manner, all packed into a soft robotic finger using the Soft Polyhedral Network design. It is generally applicable to miniature vision systems placed beneath deformable networks with a see-through design, capturing real-time images of the network's physical interactions induced by contact-based events, overlaid on the visual scene of the external environment, as demonstrated in the ablation study. We present the VBSeeThruP's capability for learning reactive grasping without using external cameras or dedicated force and torque sensors on the fingertips. Using the inpainted scene and the deformation mask, we further demonstrate the multimodal performance of the VBSeeThruP architecture to simultaneously achieve various perceptions, including but not limited to scene inpainting, object detection, depth sensing, scene segmentation, masked deformation tracking, 6D force/torque sensing, and contact event detection, all within a single sensory input from the in-finger vision markerlessly.

翻译：感官替代使生物系统能够感知通常由另一器官感知的刺激，这对物理智能体具有启发意义。内在与外在交互的多模态感知对于构建具备学习能力的智能机器人至关重要。本研究提出一种基于视觉的透视感知架构，该架构能够以无标记方式从单一视觉输入同时感知多个内在与外在模态，并通过软体多面体网络设计将其集成于软体机器人手指中。该架构普遍适用于置于可变形网络下方的微型视觉系统，其透视设计可捕获由接触事件引发的网络物理交互实时图像，并叠加于外部环境视觉场景之上，如消融实验所示。我们展示了VBSeeThruP在不使用外部摄像头或指尖专用力/扭矩传感器的情况下学习反应式抓取的能力。利用修复场景与形变掩模，我们进一步论证了VBSeeThruP架构的多模态性能，能够同时实现包括但不限于场景修复、目标检测、深度感知、场景分割、掩模形变追踪、六维力/扭矩感知及接触事件检测在内的多种感知功能，所有感知均源自手指内视觉系统的单一无标记传感输入。