Efficiently tackling multiple tasks within complex environment, such as those found in robot manipulation, remains an ongoing challenge in robotics and an opportunity for data-driven solutions, such as reinforcement learning (RL). Model-based RL, by building a dynamic model of the robot, enables data reuse and transfer learning between tasks with the same robot and similar environment. Furthermore, data gathering in robotics is expensive and we must rely on data efficient approaches such as model-based RL, where policy learning is mostly conducted on cheaper simulations based on the learned model. Therefore, the quality of the model is fundamental for the performance of the posterior tasks. In this work, we focus on improving the quality of the model and maintaining the data efficiency by performing active learning of the dynamic model during a preliminary exploration phase based on maximize information gathering. We employ Bayesian neural network models to represent, in a probabilistic way, both the belief and information encoded in the dynamic model during exploration. With our presented strategies we manage to actively estimate the novelty of each transition, using this as the exploration reward. In this work, we compare several Bayesian inference methods for neural networks, some of which have never been used in a robotics context, and evaluate them in a realistic robot manipulation setup. Our experiments show the advantages of our Bayesian model-based RL approach, with similar quality in the results than relevant alternatives with much lower requirements regarding robot execution steps. Unlike related previous studies that focused the validation solely on toy problems, our research takes a step towards more realistic setups, tackling robotic arm end-tasks.

翻译：在机器人操作等复杂环境中高效处理多个任务仍是机器人学的持续挑战，也是强化学习等数据驱动解决方案的机遇。基于模型的强化学习通过构建机器人动力学模型，能够在相同机器人和相似环境下实现数据复用与任务间迁移学习。此外，机器人领域的数据采集成本高昂，必须依赖基于模型的强化学习等数据高效方法——此类方法主要通过基于学习模型的廉价仿真进行策略学习。因此，模型质量对后续任务的性能至关重要。本研究聚焦于通过最大信息采集原则，在初步探索阶段对动力学模型进行主动学习，以提升模型质量并保持数据效率。我们采用贝叶斯神经网络模型，以概率化方式表征探索过程中动力学模型蕴含的置信度与信息量。通过提出的策略，我们能够主动估计每个状态转移的新颖性，并将其作为探索奖励。本文比较了多种神经网络贝叶斯推断方法（部分此前从未应用于机器人领域），并在逼真的机器人操作场景中评估其性能。实验表明，本研究的贝叶斯模型强化学习方法具有优势——在机器人执行步骤需求远低于同类方案的前提下，取得可比拟的结果质量。与以往仅聚焦于玩具问题的验证研究不同，本研究向更真实的场景迈出关键一步，直接处理机械臂终端任务。