Although Reinforcement Learning (RL) has shown to be capable of producing impressive results, its use is limited by the impact of its hyperparameters on performance. This often makes it difficult to achieve good results in practice. Automated RL (AutoRL) addresses this difficulty, yet little is known about the dynamics of the hyperparameter landscapes that hyperparameter optimization (HPO) methods traverse in search of optimal configurations. In view of existing AutoRL approaches dynamically adjusting hyperparameter configurations, we propose an approach to build and analyze these hyperparameter landscapes not just for one point in time but at multiple points in time throughout training. Addressing an important open question on the legitimacy of such dynamic AutoRL approaches, we provide thorough empirical evidence that the hyperparameter landscapes strongly vary over time across representative algorithms from RL literature (DQN, PPO, and SAC) in different kinds of environments (Cartpole, Bipedal Walker, and Hopper) This supports the theory that hyperparameters should be dynamically adjusted during training and shows the potential for more insights on AutoRL problems that can be gained through landscape analyses. Our code can be found at https://github.com/automl/AutoRL-Landscape

翻译：尽管强化学习（RL）已展现出产生显著结果的能力，但其应用受限于超参数对性能的影响，这往往使其在实践中难以获得良好效果。自动强化学习（AutoRL）旨在解决这一难题，然而对于超参数优化（HPO）方法在寻求最优配置时所穿越的超参数景观的动态特性，目前仍知之甚少。鉴于现有AutoRL方法会动态调整超参数配置，本文提出了一种构建与分析超参数景观的方法，该分析不仅针对单一时间点，而是贯穿训练过程中的多个时间点。针对这类动态AutoRL方法有效性的关键开放问题，我们提供了充分的实证证据，证明在RL文献中的代表性算法（DQN、PPO与SAC）以及不同类型环境（Cartpole、Bipedal Walker和Hopper）中，超参数景观会随训练时间发生显著变化。这一发现支持了训练过程中应动态调整超参数的理论，并展示了通过景观分析获取AutoRL问题更深入洞察的潜力。我们的代码可从https://github.com/automl/AutoRL-Landscape获取。