Reinforcement learning is still struggling with solving long-horizon surgical robot tasks which involve multiple steps over an extended duration of time due to the policy exploration challenge. Recent methods try to tackle this problem by skill chaining, in which the long-horizon task is decomposed into multiple subtasks for easing the exploration burden and subtask policies are temporally connected to complete the whole long-horizon task. However, smoothly connecting all subtask policies is difficult for surgical robot scenarios. Not all states are equally suitable for connecting two adjacent subtasks. An undesired terminate state of the previous subtask would make the current subtask policy unstable and result in a failed execution. In this work, we introduce value-informed skill chaining (ViSkill), a novel reinforcement learning framework for long-horizon surgical robot tasks. The core idea is to distinguish which terminal state is suitable for starting all the following subtask policies. To achieve this target, we introduce a state value function that estimates the expected success probability of the entire task given a state. Based on this value function, a chaining policy is learned to instruct subtask policies to terminate at the state with the highest value so that all subsequent policies are more likely to be connected for accomplishing the task. We demonstrate the effectiveness of our method on three complex surgical robot tasks from SurRoL, a comprehensive surgical simulation platform, achieving high task success rates and execution efficiency. Code is available at $\href{https://github.com/med-air/ViSkill}{\text{https://github.com/med-air/ViSkill}}$.

翻译：强化学习在解决涉及多步骤、持续时间长的长时域手术机器人任务中仍面临策略探索挑战。近期研究尝试通过技能链式学习应对该问题——将长时域任务分解为多个子任务以减轻探索负担，并通过时序连接子任务策略来完整执行整个长时域任务。然而在手术机器人场景中，平滑连接所有子任务策略具有较大难度。并非所有状态都适合作为连接相邻子任务的枢纽：前序子任务的非理想终止状态会导致当前子任务策略不稳定，最终造成执行失败。为此，本文提出基于价值信息的技能链式学习（ViSkill）——一种面向长时域手术机器人任务的新型强化学习框架。其核心思想是区分何种终止状态适合启动所有后续子任务策略。为实现该目标，我们引入状态价值函数，用于评估给定状态下整个任务的预期成功概率。基于该价值函数，学习链式策略来引导子任务策略在具有最高价值的状态终止，从而确保后续所有策略更易被串联以完成任务。我们在SurRoL（综合性手术仿真平台）的三种复杂手术机器人任务中验证了本方法的有效性，获得了较高的任务成功率和执行效率。代码开源地址：https://github.com/med-air/ViSkill。