In general, robotic dexterous hands are equipped with various sensors for acquiring multimodal contact information such as position, force, and pose of the grasped object. This multi-sensor-based design adds complexity to the robotic system. In contrast, vision-based tactile sensors employ specialized optical designs to enable the extraction of tactile information across different modalities within a single system. Nonetheless, the decoupling design for different modalities in common systems is often independent. Therefore, as the dimensionality of tactile modalities increases, it poses more complex challenges in data processing and decoupling, thereby limiting its application to some extent. Here, we developed a multimodal sensing system based on a vision-based tactile sensor, which utilizes visual representations of tactile information to perceive the multimodal contact information of the grasped object. The visual representations contain extensive content that can be decoupled by a deep neural network to obtain multimodal contact information such as classification, position, posture, and force of the grasped object. The results show that the tactile sensing system can perceive multimodal tactile information using only one single sensor and without different data decoupling designs for different modal tactile information, which reduces the complexity of the tactile system and demonstrates the potential for multimodal tactile integration in various fields such as biomedicine, biology, and robotics.

翻译：通常，机器人的灵巧手配备多种传感器以获取抓取物体的位置、力、姿态等多模态接触信息。这种多传感器设计增加了机器人系统的复杂性。相比之下，基于视觉的触觉传感器利用专门的光学设计，能够在单一系统内提取不同模态的触觉信息。然而，常见系统中不同模态的解耦设计往往是独立的。因此，随着触觉模态维度的增加，数据处理和解耦面临更复杂的挑战，这在一定程度上限制了其应用。本文开发了一种基于视觉触觉传感器的多模态感知系统，利用触觉信息的视觉表征来感知抓取物体的多模态接触信息。视觉表征包含丰富的内容，可通过深度神经网络进行解耦，从而获取抓取物体的分类、位置、姿态和力等多模态接触信息。结果表明，该触觉传感系统仅需单个传感器即可感知多模态触觉信息，无需针对不同模态的触觉信息设计不同的数据解耦方案，这降低了触觉系统的复杂性，并展示了多模态触觉集成在生物医学、生物学和机器人学等领域的应用潜力。