Equipping multi-fingered robots with tactile sensing is crucial for achieving the precise, contact-rich, and dexterous manipulation that humans excel at. However, relying solely on tactile sensing fails to provide adequate cues for reasoning about objects' spatial configurations, limiting the ability to correct errors and adapt to changing situations. In this paper, we present Tactile Adaptation from Visual Incentives (TAVI), a new framework that enhances tactile-based dexterity by optimizing dexterous policies using vision-based rewards. First, we use a contrastive-based objective to learn visual representations. Next, we construct a reward function using these visual representations through optimal-transport based matching on one human demonstration. Finally, we use online reinforcement learning on our robot to optimize tactile-based policies that maximize the visual reward. On six challenging tasks, such as peg pick-and-place, unstacking bowls, and flipping slender objects, TAVI achieves a success rate of 73% using our four-fingered Allegro robot hand. The increase in performance is 108% higher than policies using tactile and vision-based rewards and 135% higher than policies without tactile observational input. Robot videos are best viewed on our project website: https://see-to-touch.github.io/.

翻译：赋予多指机器人触觉传感能力，是实现人类擅长的精确、高接触灵巧操作的关键。然而，仅依赖触觉传感无法为目标空间构型推理提供充分线索，限制了机器人纠错与适应动态环境的能力。本文提出基于视觉激励的触觉适应框架（TAVI），该框架通过基于视觉奖励优化灵巧策略，增强触觉灵巧性。首先，我们采用基于对比学习的目标来学习视觉表征。其次，利用最优传输匹配方法，基于单次人类演示构建视觉表征的奖励函数。最终，我们通过在线强化学习优化基于触觉的策略，以最大化视觉奖励。在棒类物体抓取放置、碗具解堆叠、细长物体翻转等六项挑战性任务中，TAVI使用四指Allegro机械手实现了73%的成功率。其性能相较于采用触觉与视觉融合奖励的策略提升108%，相较于无触觉观测输入的策略提升135%。机器人操作视频详见项目网站：https://see-to-touch.github.io/。