Tactile sensing plays an irreplaceable role in robotic material recognition. It enables robots to distinguish material properties such as their local geometry and textures, especially for materials like textiles. However, most tactile recognition methods can only classify known materials that have been touched and trained with tactile data, yet cannot classify unknown materials that are not trained with tactile data. To solve this problem, we propose a tactile zero-shot learning framework to recognise unknown materials when they are touched for the first time without requiring training tactile samples. The visual modality, providing tactile cues from sight, and semantic attributes, giving high-level characteristics, are combined together to bridge the gap between touched classes and untouched classes. A generative model is learnt to synthesise tactile features according to corresponding visual images and semantic embeddings, and then a classifier can be trained using the synthesised tactile features of untouched materials for zero-shot recognition. Extensive experiments demonstrate that our proposed multimodal generative model can achieve a high recognition accuracy of 83.06% in classifying materials that were not touched before. The robotic experiment demo and the dataset are available at https://sites.google.com/view/multimodalzsl.

翻译：触觉传感在机器人材料识别中扮演着不可替代的角色，它使机器人能够区分材料的局部几何结构和纹理等特性，尤其对于纺织品这类材料。然而，大多数触觉识别方法仅能分类那些已被触摸过且经过触觉数据训练的材料，却无法识别未经触觉数据训练的未知材料。为解决这一问题，我们提出了一种触觉零样本学习框架，用于在首次接触未知材料时无需训练触觉样本即可进行识别。通过结合视觉模态（从视觉中提供触觉线索）和语义属性（提供高层次特征），该方法在已接触与未接触类别之间架起桥梁。我们学习了一个生成模型，根据对应的视觉图像和语义嵌入合成触觉特征，进而利用未接触材料的合成触觉特征训练分类器，实现零样本识别。大量实验表明，我们提出的多模态生成模型在对未经接触的材料进行分类时，识别准确率可达83.06%。机器人实验演示及数据集可访问 https://sites.google.com/view/multimodalzsl 获取。