We introduce a novel approach that combines tactile estimation and control for in-hand object manipulation. By integrating measurements from robot kinematics and an image-based tactile sensor, our framework estimates and tracks object pose while simultaneously generating motion plans to control the pose of a grasped object. This approach consists of a discrete pose estimator that uses the Viterbi decoding algorithm to find the most likely sequence of object poses in a coarsely discretized grid, and a continuous pose estimator-controller to refine the pose estimate and accurately manipulate the pose of the grasped object. Our method is tested on diverse objects and configurations, achieving desired manipulation objectives and outperforming single-shot methods in estimation accuracy. The proposed approach holds potential for tasks requiring precise manipulation in scenarios where visual perception is limited, laying the foundation for closed-loop behavior applications such as assembly and tool use. Please see supplementary videos for real-world demonstration at https://sites.google.com/view/texterity.

翻译：我们提出了一种新颖的方法，将触觉估计与控制相结合，用于手内物体操控。通过融合机器人运动学数据和基于图像的触觉传感器测量值，我们的框架能够估计并跟踪物体位姿，同时生成运动计划以控制抓取物体的位姿。该方法包括一个离散位姿估计器，利用维特比解码算法在粗离散化网格中寻找最可能的物体位姿序列，以及一个连续位姿估计-控制器，用于细化位姿估计并精确操控抓取物体的位姿。我们在多种物体和配置上测试了该方法，实现了期望的操控目标，并且在估计精度上优于单次方法。所提出的方法在视觉受限的场景中执行精确操控任务具有潜力，为闭环行为应用（如装配和工具使用）奠定了基础。请参见补充视频中的实际演示，网址为https://sites.google.com/view/texterity。