Deep generative models have various content creation applications such as graphic design, e-commerce, and virtual Try-on. However, current works mainly focus on synthesizing realistic visual outputs, often ignoring other sensory modalities, such as touch, which limits physical interaction with users. In this work, we leverage deep generative models to create a multi-sensory experience where users can touch and see the synthesized object when sliding their fingers on a haptic surface. The main challenges lie in the significant scale discrepancy between vision and touch sensing and the lack of explicit mapping from touch sensing data to a haptic rendering device. To bridge this gap, we collect high-resolution tactile data with a GelSight sensor and create a new visuotactile clothing dataset. We then develop a conditional generative model that synthesizes both visual and tactile outputs from a single sketch. We evaluate our method regarding image quality and tactile rendering accuracy. Finally, we introduce a pipeline to render high-quality visual and tactile outputs on an electroadhesion-based haptic device for an immersive experience, allowing for challenging materials and editable sketch inputs.

翻译：深度生成模型在平面设计、电子商务和虚拟试穿等各类内容创作中具有广泛应用。然而，现有研究主要聚焦于合成逼真的视觉输出，往往忽略触觉等其他感官模态，限制了与用户的物理交互。本文利用深度生成模型创建多感官体验——当用户在触觉表面滑动手指时，既能触摸又能看到合成的物体。其核心挑战在于视觉与触觉感知存在显著尺度差异，且缺乏从触觉传感数据到触觉渲染设备的显式映射。为弥合这一鸿沟，我们使用GelSight传感器采集高分辨率触觉数据，构建全新的视觉-触觉服装数据集，进而开发出能够根据单一草图同时合成视觉与触觉输出的条件生成模型。我们通过图像质量与触觉渲染精度对方法进行评价，最后提出一套流水线，可在基于电吸附的触觉设备上渲染高质量视觉与触觉输出，支持复杂材质及可编辑草图输入，实现沉浸式体验。