High-fidelity visuo-tactile sensing is important for precise robotic manipulation. However, most vision-based tactile sensors face a fundamental trade-off: opaque coatings enable tactile sensing but block pre-contact vision. To address this, we propose MuxGel, a spatially multiplexed sensor that captures both external visual information and contact-induced tactile signals through a single camera. By using a checkerboard coating pattern, MuxGel interleaves tactile-sensitive regions with transparent windows for external vision. This design maintains standard form factors, allowing for plug-and-play integration into GelSight-style sensors by simply replacing the gel pad. To recover full-resolution vision and tactile signals from the multiplexed inputs, we develop a U-Net-based reconstruction framework. Leveraging a sim-to-real pipeline, our model effectively decouples and restores high-fidelity tactile and visual fields simultaneously. Experiments on unseen objects demonstrate the framework's generalization and accuracy. Furthermore, we demonstrate MuxGel's utility in grasping tasks, where dual-modality feedback facilitates both pre-contact alignment and post-contact interaction. Results show that MuxGel enhances the perceptual capabilities of existing vision-based tactile sensors while maintaining compatibility with their hardware stacks. Project webpage: https://zhixianhu.github.io/muxgel/.
翻译:高保真的视觉-触觉感知对于精确的机器人操作至关重要。然而,大多数基于视觉的触觉传感器面临一个根本性的权衡:不透明涂层虽能实现触觉感知,却会阻挡接触前的视觉信息。为解决这一问题,我们提出MuxGel,一种空间复用传感器,可通过单个摄像头同时捕获外部视觉信息和接触引发的触觉信号。通过采用棋盘格涂层图案,MuxGel将触觉敏感区域与用于外部视觉的透明窗口交错排列。该设计保持了标准外形尺寸,仅需替换凝胶垫即可实现与GelSight式传感器的即插即用集成。为从复用输入中恢复全分辨率的视觉与触觉信号,我们开发了一种基于U-Net的重建框架。借助仿真到现实的训练流程,我们的模型能有效解耦并同时恢复高保真的触觉场与视觉场。在未见物体上的实验验证了该框架的泛化能力与准确性。此外,我们展示了MuxGel在抓取任务中的应用价值,其中双模态反馈同时促进了接触前的对准与接触后的交互。结果表明,MuxGel在保持与现有视觉触觉传感器硬件栈兼容性的同时,显著增强了其感知能力。项目网页:https://zhixianhu.github.io/muxgel/。