When using a tool, the grasps used for picking it up, reposing, and holding it in a suitable pose for the desired task could be distinct. Therefore, a key challenge for autonomous in-hand tool manipulation is finding a sequence of grasps that facilitates every step of the tool use process while continuously maintaining force closure and stability. Due to the complexity of modeling the contact dynamics, reinforcement learning (RL) techniques can provide a solution in this continuous space subject to highly parameterized physical models. However, these techniques impose a trade-off in adaptability and data efficiency. At test time the tool properties, desired trajectory, and desired application forces could differ substantially from training scenarios. Adapting to this necessitates more data or computationally expensive online policy updates. In this work, we apply the principles of discrete dynamic programming (DP) to augment RL performance with domain knowledge. Specifically, we first design a computationally simple approximation of our environment. We then demonstrate in physical simulation that performing tree searches (i.e., lookaheads) and policy rollouts with this approximation can improve an RL-derived grasp sequence policy with minimal additional online computation. Additionally, we show that pretraining a deep RL network with the DP-derived solution to the discretized problem can speed up policy training.

翻译：摘要：在使用工具时，用于拾取、重新放置以及将工具保持于执行任务所需合适姿态的抓取方式可能各不相同。因此，自主灵巧工具操作的一个关键挑战在于找到能促进工具使用每个步骤的抓取序列，同时持续保持力闭合与稳定性。由于接触动力学建模的复杂性，强化学习技术可在受高度参数化物理模型约束的连续空间中提供解决方案。然而，这些技术在适应性与数据效率之间存在权衡。在测试阶段，工具属性、期望轨迹及期望施力可能与训练场景存在显著差异。为此进行适应性调整需要更多数据或需要计算成本高昂的在线策略更新。本研究应用离散动态规划原理，将领域知识融入强化学习性能增强。具体而言，我们首先设计了一个计算简单的环境近似模型。随后在物理仿真中证明：利用该近似模型执行树搜索（即前瞻）与策略展开，可在最小化额外在线计算量的前提下改进基于强化学习的抓取序列策略。此外，我们展示使用离散化问题的动态规划求解结果预训练深度强化学习网络能够加速策略训练。