Dexterous telemanipulation is crucial in advancing human-robot systems, especially in tasks requiring precise and safe manipulation. However, it faces significant challenges due to the physical differences between human and robotic hands, the dynamic interaction with objects, and the indirect control and perception of the remote environment. Current approaches predominantly focus on mapping the human hand onto robotic counterparts to replicate motions, which exhibits a critical oversight: it often neglects the physical interaction with objects and relegates the interaction burden to the human to adapt and make laborious adjustments in response to the indirect and counter-intuitive observation of the remote environment. This work develops an End-Effects-Oriented Learning-based Dexterous Telemanipulation (EFOLD) framework to address telemanipulation tasks. EFOLD models telemanipulation as a Markov Game, introducing multiple end-effect features to interpret the human operator's commands during interaction with objects. These features are used by a Deep Reinforcement Learning policy to control the robot and reproduce such end effects. EFOLD was evaluated with real human subjects and two end-effect extraction methods for controlling a virtual Shadow Robot Hand in telemanipulation tasks. EFOLD achieved real-time control capability with low command following latency (delay<0.11s) and highly accurate tracking (MSE<0.084 rad).

翻译：灵巧遥操作对于推动人机系统发展至关重要，尤其在需要精确安全操控的任务中。然而，由于人手与机器人手之间的物理差异、与物体的动态交互以及对远端环境的间接控制与感知，该领域面临重大挑战。现有方法主要侧重于将人手动作映射到机器人手上以复现运动，这存在一个关键疏忽：往往忽略与物体的物理交互，并将交互负担转嫁给操作者，使其必须根据对远端环境间接且反直觉的观测进行适应性调整和繁重修正。本研究提出一种面向末端效应的学习型灵巧遥操作（EFOLD）框架来解决遥操作任务。EFOLD将遥操作建模为马尔可夫博弈，引入多种末端效应特征来解析操作者与物体交互过程中的指令。这些特征被深度强化学习策略用于控制机器人并复现相应的末端效应。EFOLD通过真实人类受试者和两种末端效应提取方法进行了评估，用于在遥操作任务中控制虚拟Shadow Robot Hand。EFOLD实现了实时控制能力，具有较低指令跟随延迟（延迟<0.11秒）和高精度跟踪性能（均方误差<0.084弧度）。