This paper presents T3: Transferable Tactile Transformers, a framework for tactile representation learning that scales across multi-sensors and multi-tasks. T3 is designed to overcome the contemporary issue that camera-based tactile sensing is extremely heterogeneous, i.e. sensors are built into different form factors, and existing datasets were collected for disparate tasks. T3 captures the shared latent information across different sensor-task pairings by constructing a shared trunk transformer with sensor-specific encoders and task-specific decoders. The pre-training of T3 utilizes a novel Foundation Tactile (FoTa) dataset, which is aggregated from several open-sourced datasets and it contains over 3 million data points gathered from 13 sensors and 11 tasks. FoTa is the largest and most diverse dataset in tactile sensing to date and it is made publicly available in a unified format. Across various sensors and tasks, experiments show that T3 pre-trained with FoTa achieved zero-shot transferability in certain sensor-task pairings, can be further fine-tuned with small amounts of domain-specific data, and its performance scales with bigger network sizes. T3 is also effective as a tactile encoder for long horizon contact-rich manipulation. Results from sub-millimeter multi-pin electronics insertion tasks show that T3 achieved a task success rate 25% higher than that of policies trained with tactile encoders trained from scratch, or 53% higher than without tactile sensing. Data, code, and model checkpoints are open-sourced at https://t3.alanz.info.

翻译：本文提出T3（可迁移触觉Transformer）——一种能够在多传感器与多任务间扩展的触觉表征学习框架。T3旨在解决当前基于摄像头的触觉传感技术面临的异构性难题：传感器采用不同的形态结构，现有数据集亦针对不同任务采集。T3通过构建包含传感器专用编码器与任务专用解码器的共享主干Transformer，捕获不同传感器-任务配对间的共享潜在信息。该框架使用新型基础触觉数据集（FoTa）进行预训练，该数据集整合了多个开源数据集，包含从13种传感器和11类任务中采集的超过300万个数据点。FoTa是迄今规模最大、多样性最丰富的触觉传感数据集，并以统一格式公开发布。跨多种传感器与任务的实验表明：基于FoTa预训练的T3在特定传感器-任务配对中实现了零样本迁移能力，可通过少量领域特定数据进一步微调，且其性能随网络规模扩大而提升。T3作为触觉编码器在长时序密集接触操作任务中同样表现优异。亚毫米级多引脚电子元件插装任务的实验结果显示：相较于从头训练的触觉编码器策略，T3的任务成功率提升25%；相较于无触觉传感的方案，成功率提升53%。数据、代码与模型检查点已开源：https://t3.alanz.info。