Tactile information is a critical tool for dexterous manipulation. As humans, we rely heavily on tactile information to understand objects in our environments and how to interact with them. We use touch not only to perform manipulation tasks but also to learn how to perform these tasks. Therefore, to create robotic agents that can learn to complete manipulation tasks at a human or super-human level of performance, we need to properly incorporate tactile information into both skill execution and skill learning. In this paper, we investigate how we can incorporate tactile information into imitation learning platforms to improve performance on manipulation tasks. We show that incorporating visuo-tactile pretraining improves imitation learning performance, not only for tactile agents (policies that use tactile information at inference), but also for non-tactile agents (policies that do not use tactile information at inference). For these non-tactile agents, pretraining with tactile information significantly improved performance (for example, improving the accuracy on USB plugging from 20% to 85%), reaching a level on par with visuo-tactile agents, and even surpassing them in some cases. For demonstration videos and access to our codebase, see the project website: https://sites.google.com/andrew.cmu.edu/visuo-tactile-pretraining

翻译：触觉信息是实现灵巧操作的关键工具。作为人类，我们高度依赖触觉信息来理解环境中的物体及其交互方式。我们不仅利用触觉执行操作任务，还通过触觉学习如何完成这些任务。因此，为创建能够学习达到人类或超人类操作水平的机器人智能体，必须将触觉信息有效整合到技能执行与技能学习过程中。本文研究如何将触觉信息融入模仿学习框架以提升操作任务性能。实验表明，引入视觉-触觉预训练不仅能提升触觉智能体（在推理阶段使用触觉信息的策略）的模仿学习性能，对非触觉智能体（推理阶段不使用触觉信息的策略）同样具有显著改善效果。对于非触觉智能体，触觉预训练使其性能大幅提升（例如USB插拔任务准确率从20%提高至85%），达到与视觉-触觉智能体相当的水平，甚至在部分场景中实现反超。演示视频及代码库访问详见项目网站：https://sites.google.com/andrew.cmu.edu/visuo-tactile-pretraining