To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object's pose and shape. The status quo for in-hand perception primarily employs vision, and restricts to tracking a priori known objects. Moreover, visual occlusion of objects in-hand is imminent during manipulation, preventing current systems to push beyond tasks without occlusion. We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We study multimodal in-hand perception in simulation and the real-world, interacting with different objects via a proprioception-driven policy. Our experiments show final reconstruction F-scores of $81$% and average pose drifts of $4.7\,\text{mm}$, further reduced to $2.3\,\text{mm}$ with known CAD models. Additionally, we observe that under heavy visual occlusion we can achieve up to $94$% improvements in tracking compared to vision-only methods. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step towards benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone towards advancing robot dexterity. Videos can be found on our project website https://suddhu.github.io/neural-feels/

翻译：为了达到人类水平的灵巧性，机器人必须从多模态感知中推断空间意识，以推理接触交互。在操作陌生物体的手内操作过程中，这种空间意识涉及估计物体的姿态和形状。当前手内感知的主流方法主要依赖视觉，且局限于跟踪已知物体。此外，操作过程中物体在手中的视觉遮挡不可避免，阻碍了现有系统突破无遮挡任务。我们在多指手上结合视觉和触觉传感，以估计手内操作中物体的姿态和形状。我们的方法NeuralFeels通过在线学习神经场来编码物体几何结构，并通过优化位姿图问题联合跟踪物体。我们在仿真和真实世界中研究了多模态手内感知，通过本体感知驱动策略与不同物体交互。实验表明，最终重建F分数达到81%，平均位姿漂移为4.7毫米，在已知CAD模型下进一步降低至2.3毫米。此外，我们观察到在严重视觉遮挡下，相较于纯视觉方法，跟踪性能可提升高达94%。我们的结果表明，在手内操作中，触觉至少能优化视觉估计，且最佳情况下能消除视觉歧义。我们发布了包含70次实验的评估数据集FeelSight，旨在为该领域建立基准。这种由多模态感知驱动的神经表示可作为推进机器人灵巧性的感知骨架。视频可在项目网站 https://suddhu.github.io/neural-feels/ 查看。