Robot arms should be able to learn new tasks. One framework here is reinforcement learning, where the robot is given a reward function that encodes the task, and the robot autonomously learns actions to maximize its reward. Existing approaches to reinforcement learning often frame this problem as a Markov decision process, and learn a policy (or a hierarchy of policies) to complete the task. These policies reason over hundreds of fine-grained actions that the robot arm needs to take: e.g., moving slightly to the right or rotating the end-effector a few degrees. But the manipulation tasks that we want robots to perform can often be broken down into a small number of high-level motions: e.g., reaching an object or turning a handle. In this paper we therefore propose a waypoint-based approach for model-free reinforcement learning. Instead of learning a low-level policy, the robot now learns a trajectory of waypoints, and then interpolates between those waypoints using existing controllers. Our key novelty is framing this waypoint-based setting as a sequence of multi-armed bandits: each bandit problem corresponds to one waypoint along the robot's motion. We theoretically show that an ideal solution to this reformulation has lower regret bounds than standard frameworks. We also introduce an approximate posterior sampling solution that builds the robot's motion one waypoint at a time. Results across benchmark simulations and two real-world experiments suggest that this proposed approach learns new tasks more quickly than state-of-the-art baselines. See videos here: https://youtu.be/MMEd-lYfq4Y

翻译：机器人手臂应能学习新任务。强化学习是一种实现框架：通过为机器人设定编码任务的奖励函数，使其自主学习最大化奖励的动作。现有强化学习方法通常将问题建模为马尔可夫决策过程，并学习执行任务所需的策略（或策略层级）。这些策略需处理数百个精细动作——例如微小右移或末端执行器旋转数度——但目标操作任务往往可分解为少量高阶动作（如抓取物体或旋转手柄）。为此，本文提出一种基于航路点的无模型强化学习方法：机器人不再学习低层级策略，而是学习一组航路点轨迹，并通过现有控制器在这些航路点间插值。核心创新在于将航路点框架建模为多臂老虎机序列——每个老虎机问题对应机器人运动轨迹的一个航路点。理论证明表明，该重构的理想解具有更低的遗憾界。同时引入近似后验采样方法，逐步构建机器人运动路径。基准仿真与两项真实实验结果表明，所提方法在任务学习速度上优于现有最先进基线。相关视频见 https://youtu.be/MMEd-lYfq4Y