How to reduce compute and memory requirements of neural networks (NNs) without sacrificing performance? Many recent works use sparse Mixtures of Experts (MoEs) to build resource-efficient large language models (LMs). Here we introduce several novel perspectives on MoEs, presenting a general framework that unifies various methods to approximate two-layer NNs (e.g., feedforward blocks of Transformers), including product-key memories (PKMs). Leveraging insights from this framework, we propose methods to improve both MoEs and PKMs. Unlike prior work that compares MoEs with dense baselines under the compute-equal condition, our evaluation condition is parameter-equal, which is crucial to properly evaluate LMs. We show that our MoEs are competitive with the dense Transformer-XL on both the WikiText-103 and enwiki8 datasets at two different scales, while being much more resource efficient. This demonstrates that MoEs are relevant not only to extremely large LMs but also to any-scale resource-efficient LMs. Our code is public.

翻译：如何在不牺牲性能的前提下降低神经网络的计算和内存需求？近期许多研究采用稀疏专家混合体构建资源高效的大语言模型。本文引入若干关于专家混合体的新视角，提出一个统一多种双层神经网络（如Transformer前馈模块）近似方法的通用框架——包括乘积键记忆机制。基于该框架的洞察，我们提出了改进专家混合体和乘积键记忆的方法。不同于先前在计算等量条件下比较专家混合体与密集基线的研究，我们的评估采用参数等量条件——这对合理评估语言模型至关重要。实验表明，在WikiText-103和enwiki8两个数据集的两个不同规模上，我们的专家混合体模型与密集Transformer-XL性能相当，同时资源效率显著更高。这证明了专家混合体不仅适用于超大规模语言模型，对任意规模的资源高效语言模型均有重要意义。我们的代码已开源。