Simulating tactile perception could potentially leverage the learning capabilities of robotic systems in manipulation tasks. However, the reality gap of simulators for high-resolution tactile sensors remains large. Models trained on simulated data often fail in zero-shot inference and require fine-tuning with real data. In addition, work on high-resolution sensors commonly focus on ones with flat surfaces while 3D round sensors are essential for dexterous manipulation. In this paper, we propose a bi-directional Generative Adversarial Network (GAN) termed SightGAN. SightGAN relies on the early CycleGAN while including two additional loss components aimed to accurately reconstruct background and contact patterns including small contact traces. The proposed SightGAN learns real-to-sim and sim-to-real processes over difference images. It is shown to generate real-like synthetic images while maintaining accurate contact positioning. The generated images can be used to train zero-shot models for newly fabricated sensors. Consequently, the resulted sim-to-real generator could be built on top of the tactile simulator to provide a real-world framework. Potentially, the framework can be used to train, for instance, reinforcement learning policies of manipulation tasks. The proposed model is verified in extensive experiments with test data collected from real sensors and also shown to maintain embedded force information within the tactile images.

翻译：模拟触觉感知有望提升机器人系统在操作任务中的学习能力。然而，高分辨率触觉传感器模拟器与现实环境间的差距依然显著。基于模拟数据训练的模型在零样本推理中常表现不佳，需通过真实数据进行微调。此外，现有高分辨率传感器研究多聚焦于平面结构，而三维球形传感器对灵巧操作至关重要。本文提出一种双向生成对抗网络——SightGAN。该网络基于早期CycleGAN架构，额外引入两项损失函数，旨在精确重建背景与接触模式（含微小接触痕迹）。SightGAN通过差分图像学习模拟-真实与真实-模拟的双向映射过程。实验证明，该方法能生成具有真实感且保持准确保触位置的合成图像，可用于训练新制传感器的零样本模型。由此产生的模拟-真实生成器可叠加于触觉模拟器之上，构建面向真实世界的框架。该框架理论上可支持如操作任务的强化学习策略训练。通过真实传感器采集数据的广泛实验验证，该模型生成的触觉图像能有效保留嵌入的力信息。