We introduce a novel approach that combines tactile estimation and control for in-hand object manipulation. By integrating measurements from robot kinematics and an image-based tactile sensor, our framework estimates and tracks object pose while simultaneously generating motion plans in a receding horizon fashion to control the pose of a grasped object. This approach consists of a discrete pose estimator that tracks the most likely sequence of object poses in a coarsely discretized grid, and a continuous pose estimator-controller to refine the pose estimate and accurately manipulate the pose of the grasped object. Our method is tested on diverse objects and configurations, achieving desired manipulation objectives and outperforming single-shot methods in estimation accuracy. The proposed approach holds potential for tasks requiring precise manipulation and limited intrinsic in-hand dexterity under visual occlusion, laying the foundation for closed-loop behavior in applications such as regrasping, insertion, and tool use. Please see https://sites.google.com/view/texterity for videos of real-world demonstrations.

翻译：我们提出了一种结合触觉估计与控制的新型人手内物体操控方法。通过融合机器人运动学测量与基于图像的触觉传感器数据，本框架能同步估计并追踪物体位姿，同时以滚动时域方式生成运动规划以控制抓取物体的位姿。该方法由离散位姿估计器与连续位姿估计-控制器组成：前者在粗离散化网格中追踪最可能的物体位姿序列，后者则精细化位姿估计并精确操控被抓取物体的位姿。本方法在多种物体与配置下完成测试，实现了预期操控目标，且位姿估计精度优于单次估计算法。该方案为视觉遮挡条件下需精密度操控且内在灵巧性有限的场景提供了潜在解决方案，为再抓取、插入与工具使用等应用中的闭环行为奠定了基础。实际演示视频请见 https://sites.google.com/view/texterity。