The recent rapid progress in (self) supervised learning models is in large part predicted by empirical scaling laws: a model's performance scales proportionally to its size. Analogous scaling laws remain elusive for reinforcement learning domains, however, where increasing the parameter count of a model often hurts its final performance. In this paper, we demonstrate that incorporating Mixture-of-Expert (MoE) modules, and in particular Soft MoEs (Puigcerver et al., 2023), into value-based networks results in more parameter-scalable models, evidenced by substantial performance increases across a variety of training regimes and model sizes. This work thus provides strong empirical evidence towards developing scaling laws for reinforcement learning.

翻译：近期（自）监督学习模型的快速进展在很大程度上可由经验缩放定律预测：模型的性能与其规模成比例扩展。然而，在强化学习领域，类似的缩放定律仍然难以捉摸，增加模型的参数数量往往会损害其最终性能。本文证明，将专家混合（MoE）模块，特别是软专家混合（Soft MoEs，Puigcerver等人，2023）融入基于价值的网络中，可产生更具参数可扩展性的模型，这体现在多种训练机制和模型规模下性能的显著提升。因此，本研究为发展强化学习的缩放定律提供了强有力的实证依据。