We introduce a novel approach that combines tactile estimation and control for in-hand object manipulation. By integrating measurements from robot kinematics and an image-based tactile sensor, our framework estimates and tracks object pose while simultaneously generating motion plans in a receding horizon fashion to control the pose of a grasped object. This approach consists of a discrete pose estimator that tracks the most likely sequence of object poses in a coarsely discretized grid, and a continuous pose estimator-controller to refine the pose estimate and accurately manipulate the pose of the grasped object. Our method is tested on diverse objects and configurations, achieving desired manipulation objectives and outperforming single-shot methods in estimation accuracy. The proposed approach holds potential for tasks requiring precise manipulation and limited intrinsic in-hand dexterity under visual occlusion, laying the foundation for closed-loop behavior in applications such as regrasping, insertion, and tool use. Please see https://sites.google.com/view/texterity for videos of real-world demonstrations.

翻译：我们提出了一种新颖方法，融合了触觉估计与控制技术以实现手中物体的灵巧操作。通过集成机器人运动学测量与基于图像的触觉传感器，本框架能够估计并跟踪物体位姿，同时以滚动时域方式生成运动规划，从而控制抓取物体的位姿。该方法包含一个离散位姿估计器，用于在粗粒度离散网格中追踪最可能的物体位姿序列，以及一个连续位姿估计-控制器，用于优化位姿估计并精确操控被抓物体的位姿。本方法在多种物体与构型下进行了测试，成功实现了预期操控目标，并在估计精度上优于单次估计方法。该技术在处理需要精细操控且视觉遮挡下内禀灵巧性受限的任务中具有潜力，为重新抓取、插入操作及工具使用等场景中的闭环行为奠定了技术基础。详见 https://sites.google.com/view/texterity 观看实际演示视频。