The current practice of dexterous manipulation generally relies on a single wrist-mounted view, which is often occluded and limits performance on tasks requiring multi-view perception. In this work, we present FingerViP, a learning system that utilizes a visuomotor policy with fingertip visual perception for dexterous manipulation. Specifically, we design a vision-enhanced fingertip module with an embedded miniature camera and install the modules on each finger of a multi-fingered hand. The fingertip cameras substantially improve visual perception by providing comprehensive, multi-view feedback of both the hand and its surrounding environment. Building on the integrated fingertip modules, we develop a diffusion-based whole-body visuomotor policy conditioned on a third-view camera and multi-view fingertip vision, which effectively learns complex manipulation skills directly from human demonstrations. To improve view-proprioception alignment and contact awareness, each fingertip visual feature is augmented with its corresponding camera pose encoding and per-finger joint-current encoding. We validate the effectiveness of the multi-view fingertip vision and demonstrate the robustness and adaptability of FingerViP on various challenging real-world tasks, including pressing buttons inside a confined box, retrieving sticks from an unstable support, retrieving objects behind an occluding curtain, and performing long-horizon cabinet opening and object retrieval, achieving an overall success rate of 80.8%. All hardware designs and code will be fully open-sourced.

翻译：当前灵巧操作的实践通常依赖单个腕部安装的视角，该视角常被遮挡，限制了需要多视角感知的任务性能。本文提出FingerViP，一种利用基于指尖视觉感知的视觉运动策略进行灵巧操作的学习系统。具体而言，我们设计了一种配备嵌入式微型摄像头的视觉增强指尖模块，并将其安装于多指手的每个手指上。指尖摄像头通过提供手部及周围环境的全面多视角反馈，显著提升了视觉感知能力。基于集成的指尖模块，我们开发了一种扩散式全身视觉运动策略，该策略以第三视角摄像头和多视角指尖视觉为条件，能够直接从人类示范中有效学习复杂操作技能。为改善视角-本体感知对齐与接触意识，每个指尖视觉特征均辅以对应的相机位姿编码和手指关节电流编码。我们验证了多视角指尖视觉的有效性，并展示了FingerViP在多种具有挑战性的真实世界任务中的鲁棒性和适应性，包括在封闭箱内按压按钮、从不稳定支架中取出棍子、从遮挡帘后取物，以及执行长时程柜门开启与物体取回任务，总体成功率达80.8%。所有硬件设计与代码将完全开源。