Large language models (LLMs) have significantly advanced generative applications in natural language processing (NLP). Recent trends in model architectures revolve around efficient variants of transformers or state-space/gated-recurrent models (SSMs, GRMs). However, prevailing SSM/GRM-based methods often emulate only a single attention head, potentially limiting their expressiveness. In this work, we propose MossNet, a novel mixture-of-state-space-experts architecture that emulates a linear multi-head attention (MHA). MossNet leverages a mixture-of-experts (MoE) implementation not only in channel-mixing multi-layered perceptron (MLP) blocks but also in the time-mixing SSM kernels to realize multiple "attention heads." Extensive experiments on language modeling and downstream evaluations show that MossNet outperforms both transformer- and SSM-based architectures of similar model size and data budgets. Larger variants of MossNet, trained on trillions of tokens, further confirm its scalability and superior performance. In addition, real-device profiling on a Samsung Galaxy S24 Ultra and an Nvidia A100 GPU demonstrate favorable runtime speed and resource usage compared to similarly sized baselines. Our results suggest that MossNet is a compelling new direction for efficient, high-performing recurrent LLM architectures.

翻译：大型语言模型（LLMs）显著推动了自然语言处理（NLP）中生成式应用的发展。当前模型架构的趋势主要围绕Transformer的高效变体或状态空间/门控循环模型（SSMs、GRMs）展开。然而，主流的基于SSM/GRM的方法通常仅模拟单个注意力头，这可能限制了其表达能力。本研究提出MossNet，一种新颖的状态空间专家混合架构，能够模拟线性多头注意力（MHA）。MossNet不仅在通道混合的多层感知机（MLP）模块中采用专家混合（MoE）实现，还在时间混合的SSM核中应用MoE，以实现多个“注意力头”。在语言建模和下游任务评估上的大量实验表明，MossNet在相似模型规模和数据预算下，性能优于基于Transformer和SSM的架构。在数万亿token上训练的大规模MossNet变体进一步证实了其可扩展性和卓越性能。此外，在三星Galaxy S24 Ultra和Nvidia A100 GPU上的实际设备性能分析显示，与规模相近的基线模型相比，MossNet在运行速度和资源使用方面表现更优。我们的结果表明，MossNet为高效、高性能的循环LLM架构提供了一个引人注目的新方向。