We introduce a novel approach that combines tactile estimation and control for in-hand object manipulation. By integrating measurements from robot kinematics and an image-based tactile sensor, our framework estimates and tracks object pose while simultaneously generating motion plans to control the pose of a grasped object. This approach consists of a discrete pose estimator that uses the Viterbi decoding algorithm to find the most likely sequence of object poses in a coarsely discretized grid, and a continuous pose estimator-controller to refine the pose estimate and accurately manipulate the pose of the grasped object. Our method is tested on diverse objects and configurations, achieving desired manipulation objectives and outperforming single-shot methods in estimation accuracy. The proposed approach holds potential for tasks requiring precise manipulation in scenarios where visual perception is limited, laying the foundation for closed-loop behavior applications such as assembly and tool use. Please see supplementary videos for real-world demonstration at https://sites.google.com/view/texterity.

翻译：我们提出了一种新颖的方法，结合触觉估计与控制以进行手中物体操作。通过整合机器人运动学和基于图像的触觉传感器测量数据，我们的框架能够估计并跟踪物体位姿，同时生成运动规划以控制被抓取物体的位姿。该方法包括一个离散位姿估计器，利用维特比解码算法在粗离散化网格中找出最可能的物体位姿序列；以及一个连续位姿估计-控制器，用于细化位姿估计并精确操控被抓取物体的位姿。我们的方法在多种物体和配置上进行了测试，实现了预期的操作目标，并在估计精度上优于单次估计方法。所提出的方法在视觉感知受限的场景中具有精确操作任务的潜力，为闭环行为应用（如装配和工具使用）奠定了基础。请参见补充视频以获取真实世界演示：https://sites.google.com/view/texterity。