The costly self-attention layers in modern Transformers require memory and compute quadratic in sequence length. Existing approximation methods usually underperform and fail to obtain significant speedups in practice. Here we present SwitchHead - a novel method that reduces both compute and memory requirements and achieves wall-clock speedup, while matching the language modeling performance of baseline Transformers with the same parameter budget. SwitchHead uses Mixture-of-Experts (MoE) layers for the value and output projections and requires 4 to 8 times fewer attention matrices than standard Transformers. Our novel attention can also be combined with MoE MLP layers, resulting in an efficient fully-MoE "SwitchAll" Transformer model. Our code is public.

翻译：现代Transformer中昂贵的自注意力层需要与序列长度呈二次增长的内存和计算开销。现有近似方法通常性能不佳，且在实践中难以实现显著加速。本文提出SwitchHead——一种新颖方法，在保持与基线Transformer相同参数预算的语言建模性能的同时，可降低计算和内存需求并实现实际加速。该方法在价值和输出投影层采用混合专家（MoE）结构，所需注意力矩阵数量仅为标准Transformer的1/4至1/8。这种新型注意力机制还可与MoE MLP层结合，构建出高效的纯MoE型"SwitchAll"Transformer模型。我们的代码已公开。