The recent rapid progress in (self) supervised learning models is in large part predicted by empirical scaling laws: a model's performance scales proportionally to its size. Analogous scaling laws remain elusive for reinforcement learning domains, however, where increasing the parameter count of a model often hurts its final performance. In this paper, we demonstrate that incorporating Mixture-of-Expert (MoE) modules, and in particular Soft MoEs (Puigcerver et al., 2023), into value-based networks results in more parameter-scalable models, evidenced by substantial performance increases across a variety of training regimes and model sizes. This work thus provides strong empirical evidence towards developing scaling laws for reinforcement learning.

翻译：近期（自）监督学习模型的快速发展在很大程度上得益于经验性的缩放定律：模型性能与其规模成比例增长。然而，类似的缩放定律在强化学习领域仍难以实现——在该领域，增加模型的参数数量反而常常损害其最终性能。本文证明，将混合专家（MoE）模块，特别是软混合专家（Soft MoE, Puigcerver 等人，2023）集成到基于价值的网络中，能够产生更具参数可扩展性的模型，这在多种训练范式和模型规模下均表现出显著的性能提升。因此，本研究为开发强化学习的缩放定律提供了强有力的实证依据。