Tactile sensing is significant for robotics since it can obtain physical contact information during manipulation. To capture multimodal contact information within a compact framework, we designed a novel sensor called ViTacTip, which seamlessly integrates both tactile and visual perception capabilities into a single, integrated sensor unit. ViTacTip features a transparent skin to capture fine features of objects during contact, which can be known as the see-through-skin mechanism. In the meantime, the biomimetic tips embedded in ViTacTip can amplify touch motions during tactile perception. For comparative analysis, we also fabricated a ViTac sensor devoid of biomimetic tips, as well as a TacTip sensor with opaque skin. Furthermore, we develop a Generative Adversarial Network (GAN)-based approach for modality switching between different perception modes, effectively alternating the emphasis between vision and tactile perception modes. We conducted a performance evaluation of the proposed sensor across three distinct tasks: i) grating identification, ii) pose regression, and iii) contact localization and force estimation. In the grating identification task, ViTacTip demonstrated an accuracy of 99.72%, surpassing TacTip, which achieved 94.60%. It also exhibited superior performance in both pose and force estimation tasks with the minimum error of 0.08mm and 0.03N, respectively, in contrast to ViTac's 0.12mm and 0.15N. Results indicate that ViTacTip outperforms single-modality sensors.

翻译：触觉感知对于机器人技术至关重要，因为它能在操作过程中获取物理接触信息。为了在紧凑的框架内捕捉多模态接触信息，我们设计了一种名为ViTacTip的新型传感器，该传感器将触觉与视觉感知能力无缝集成于单个一体化传感器单元中。ViTacTip采用透明皮肤设计，在接触过程中可捕捉物体的精细特征，这一机制被称为“透视皮肤”机制。同时，ViTacTip中嵌入的仿生尖端能够在触觉感知过程中放大触觉运动。为了进行对比分析，我们还制造了不含仿生尖端的ViTac传感器，以及具有不透明皮肤的TacTip传感器。此外，我们开发了一种基于生成对抗网络（GAN）的方法，用于不同感知模式间的模态切换，从而有效交替视觉与触觉感知模式的侧重点。我们通过三个不同任务对所提出的传感器进行了性能评估：(i) 光栅识别；(ii) 姿态回归；(iii) 接触定位与力估计。在光栅识别任务中，ViTacTip的准确率达到99.72%，超越了达到94.60%的TacTip。在姿态与力估计任务中，ViTacTip同样表现出更优性能，其最小误差分别为0.08毫米和0.03牛，而ViTac对应的误差为0.12毫米和0.15牛。结果表明，ViTacTip优于单模态传感器。