This paper introduces the TactiMesh Teleoperator Interface (TTI), a novel predictive visual and haptic system designed explicitly for human-in-the-loop robot control using a head-mounted display (HMD). By employing simultaneous localization and mapping (SLAM)in tandem with a space carving method (CARV), TTI creates a real time 3D surface mesh of remote environments from an RGB camera mounted on a Barrett WAM arm. The generated mesh is integrated into a physics simulator, featuring a digital twin of the WAM robot arm to create a virtual environment. In this virtual environment, TTI provides haptic feedback directly in response to the operator's movements, eliminating the problem with delayed response from the haptic follower robot. Furthermore, texturing the 3D mesh with keyframes from SLAM allows the operator to control the viewpoint of their Head Mounted Display (HMD) independently of the arm-mounted robot camera, giving a better visual immersion and improving manipulation speed. Incorporating predictive visual and haptic feedback significantly improves teleoperation in applications such as search and rescue, inspection, and remote maintenance.

翻译：本文介绍了一种新颖的预测性视觉与触觉系统——触觉网格遥操作界面（TTI），该系统专为基于头戴式显示器（HMD）的人机协同机器人控制而设计。通过将同步定位与建图（SLAM）技术与空间雕刻方法（CARV）相结合，TTI利用安装在Barrett WAM机械臂上的RGB摄像头，实时生成远程环境的三维表面网格。生成的网格被集成到物理仿真器中，其中包含WAM机械臂的数字孪生体，从而构建出一个虚拟环境。在该虚拟环境中，TTI能够直接根据操作者的动作提供触觉反馈，解决了触觉跟随机器人响应延迟的问题。此外，利用SLAM关键帧对三维网格进行纹理映射，使得操作者能够独立于机械臂搭载的机器人摄像头来控制其头戴式显示器（HMD）的视角，从而提供更佳的视觉沉浸感并提升操作速度。引入预测性视觉与触觉反馈显著提升了在搜救、巡检和远程维护等应用中的遥操作性能。