Large Language Models (LLMs) have achieved significant success in various natural language processing tasks, but how wireless communications can support LLMs has not been extensively studied. In this paper, we propose a wireless distributed LLMs paradigm based on Mixture of Experts (MoE), named WDMoE, deploying LLMs collaboratively across edge servers of base station (BS) and mobile devices in the wireless communications system. Specifically, we decompose the MoE layer in LLMs by deploying the gating network and the preceding neural network layer at BS, while distributing the expert networks across the devices. This arrangement leverages the parallel capabilities of expert networks on distributed devices. Moreover, to overcome the instability of wireless communications, we design an expert selection policy by taking into account both the performance of the model and the end-to-end latency, which includes both transmission delay and inference delay. Evaluations conducted across various LLMs and multiple datasets demonstrate that WDMoE not only outperforms existing models, such as Llama 2 with 70 billion parameters, but also significantly reduces end-to-end latency.

翻译：大型语言模型在各种自然语言处理任务中取得了显著成功，但无线通信如何支持大型语言模型尚未得到广泛研究。本文提出了一种基于混合专家模型的无线分布式大型语言模型范式，命名为WDMoE，该范式在无线通信系统中，将大型语言模型协同部署在基站边缘服务器和移动设备上。具体而言，我们通过将门控网络和前一神经网络层部署在基站，同时将专家网络分布到各个设备上，实现大型语言模型中混合专家层的分解。这种部署方式利用了分布式设备上专家网络的并行处理能力。此外，为克服无线通信的不稳定性，我们设计了一种专家选择策略，该策略综合考虑了模型性能和端到端延迟（包括传输延迟和推理延迟）。在多种大型语言模型和多个数据集上的评估表明，WDMoE不仅优于现有模型（如具有700亿参数的Llama 2），还显著降低了端到端延迟。