Early stopping based on the validation set performance is a popular approach to find the right balance between under- and overfitting in the context of supervised learning. However, in reinforcement learning, even for supervised sub-problems such as world model learning, early stopping is not applicable as the dataset is continually evolving. As a solution, we propose a new general method that dynamically adjusts the update to data (UTD) ratio during training based on under- and overfitting detection on a small subset of the continuously collected experience not used for training. We apply our method to DreamerV2, a state-of-the-art model-based reinforcement learning algorithm, and evaluate it on the DeepMind Control Suite and the Atari $100$k benchmark. The results demonstrate that one can better balance under- and overestimation by adjusting the UTD ratio with our approach compared to the default setting in DreamerV2 and that it is competitive with an extensive hyperparameter search which is not feasible for many applications. Our method eliminates the need to set the UTD hyperparameter by hand and even leads to a higher robustness with regard to other learning-related hyperparameters further reducing the amount of necessary tuning.

翻译：基于验证集性能的早停法是监督学习中平衡欠拟合与过拟合的常用方法。但在强化学习中，即使是世界模型学习这类监督性子问题，由于数据集持续变化，早停法并不适用。为此，我们提出一种通用新方法：基于持续收集但未用于训练的小部分经验，通过检测欠拟合与过拟合状态，动态调整训练过程中的更新-数据比率。我们将该方法应用于最先进的基于模型的强化学习算法DreamerV2，并在DeepMind Control套件与Atari $100$k基准上评估。结果表明，与DreamerV2默认设置相比，我们的方法能通过调整更新-数据比率更好地平衡欠估计与过估计，且其性能可与许多应用中不可行的大规模超参数搜索相媲美。该方法不仅消除了手动设置更新-数据比率超参数的需求，还增强了对其他学习相关超参数的鲁棒性，进一步减少了必要调优量。