For robot manipulation, a complete and accurate object shape is desirable. Here, we present a method that combines visual and haptic reconstruction in a closed-loop pipeline. From an initial viewpoint, the object shape is reconstructed using an implicit surface deep neural network. The location with highest uncertainty is selected for haptic exploration, the object is touched, the new information from touch and a new point cloud from the camera are added, object position is re-estimated and the cycle is repeated. We extend Rustler et al. (2022) by using a new theoretically grounded method to determine the points with highest uncertainty, and we increase the yield of every haptic exploration by adding not only the contact points to the point cloud but also incorporating the empty space established through the robot movement to the object. Additionally, the solution is compact in that the jaws of a closed two-finger gripper are directly used for exploration. The object position is re-estimated after every robot action and multiple objects can be present simultaneously on the table. We achieve a steady improvement with every touch using three different metrics and demonstrate the utility of the better shape reconstruction in grasping experiments on the real robot. On average, grasp success rate increases from 63.3% to 70.4% after a single exploratory touch and to 82.7% after five touches. The collected data and code are publicly available (https://osf.io/j6rkd/, https://github.com/ctu-vras/vishac)

翻译：在机器人操作中，完整且准确的物体形状是可取的。本文提出了一种在闭环流程中结合视觉与触觉重建的方法。从初始视角出发，使用隐式表面深度神经网络重建物体形状。选择不确定性最高的位置进行触觉探索，触摸物体后，将来自触觉的新信息与相机获取的新点云一同添加，重新估计物体位置，并重复此循环。我们在Rustler等人（2022）的研究基础上，采用了一种新的理论指导方法来确定不确定性最高的点，并通过不仅将接触点添加至点云，同时将机器人向物体移动过程中所确定的空白空间也纳入其中，从而提高了每次触觉探索的收益。此外，该方案设计紧凑，可直接使用闭合的两指夹爪进行探索。每次机器人动作后均重新估计物体位置，且桌面上可同时存在多个物体。我们使用三种不同的指标观察到每次触摸后性能的稳步提升，并在真实机器人上的抓取实验中证明了更好形状重建的实用性。平均而言，抓取成功率从63.3%提升至单次探索性触摸后的70.4%，并在五次触摸后达到82.7%。所收集的数据和代码已公开（https://osf.io/j6rkd/, https://github.com/ctu-vras/vishac）。