How to reduce compute and memory requirements of neural networks (NNs) without sacrificing performance? Many recent works use sparse Mixtures of Experts (MoEs) to build resource-efficient large language models (LMs). Here we introduce several novel perspectives on MoEs, presenting a general framework that unifies various methods to approximate two-layer NNs (e.g., feedforward blocks of Transformers), including product-key memories (PKMs). Leveraging insights from this framework, we propose methods to improve both MoEs and PKMs. Unlike prior work that compares MoEs with dense baselines under the compute-equal condition, our evaluation condition is parameter-equal, which is crucial to properly evaluate LMs. We show that our MoEs are competitive with the dense Transformer-XL on both the WikiText-103 and enwiki8 datasets at two different scales, while being much more resource efficient. This demonstrates that MoEs are relevant not only to extremely large LMs but also to any-scale resource-efficient LMs. Our code is public.

翻译：如何在不牺牲性能的前提下降低神经网络（NNs）的计算开销与内存需求？近期许多工作采用稀疏专家混合（MoEs）来构建资源高效的大型语言模型（LMs）。本文提出关于MoEs若干新颖视角，构建统一各类两层神经网络（如Transformer的前馈模块）近似方法的通用框架（含乘积键记忆PKMs）。基于该框架的洞见，我们提出改进MoEs与PKMs的方法。不同于以往在计算量相等条件下对比MoEs与密集基线的工作，我们的评估条件设定为参数数量相等——这对合理评估LMs至关重要。实验表明，在WikiText-103和enwiki8两个数据集上，我们的MoEs在不同规模下均与密集Transformer-XL性能相当，同时资源效率显著提升。这证明MoEs不仅适用于超大规模LMs，对任意规模的资源高效LMs同样具有价值。我们的代码已开源。