As the demand for mobile robots continues to increase, social navigation has emerged as a critical task, driving active research into deep reinforcement learning (RL) approaches. However, because pedestrian dynamics and social conventions vary widely across different regions, simulations cannot easily encompass all possible real-world scenarios. Real-world RL, in which agents learn while operating directly in physical environments, presents a promising solution to this issue. Nevertheless, this approach faces significant challenges, particularly regarding constrained computational resources on edge devices and learning efficiency. In this study, we propose incremental residual RL (IRRL). This method integrates incremental learning, which is a lightweight process that operates without a replay buffer or batch updates, with residual RL, which enhances learning efficiency by training only on the residuals relative to a base policy. Through the simulation experiments, we demonstrated that, despite lacking a replay buffer, IRRL achieved performance comparable to those of conventional replay buffer-based methods and outperformed existing incremental learning approaches. Furthermore, the real-world experiments confirmed that IRRL can enable robots to effectively adapt to previously unseen environments through the real-world learning.

翻译：随着移动机器人需求的不断增长，社交导航已成为一项关键任务，驱动着对深度强化学习方法的积极研究。然而，由于行人动态和社会习俗在不同地区差异显著，仿真难以涵盖所有可能的真实世界场景。智能体直接在物理环境中运行并同时学习的真实世界强化学习，为解决这一问题提供了有前景的方案。尽管如此，该方法仍面临重大挑战，尤其是边缘设备上有限的计算资源以及学习效率问题。在本研究中，我们提出了增量式残差强化学习（IRRL）。该方法将增量学习（一种无需重放缓冲区或批量更新的轻量级过程）与残差强化学习相结合，后者通过仅训练相对于基础策略的残差来提升学习效率。通过仿真实验，我们证明尽管缺乏重放缓冲区，IRRL仍能达到与基于重放缓冲区的常规方法相当的性能，并优于现有增量学习方法。此外，真实世界实验证实，IRRL能使机器人通过真实世界学习有效适应未曾见过的环境。