This position paper investigates the integration of Differential Privacy (DP) in the training of Mixture of Experts (MoE) models within the field of natural language processing. As Large Language Models (LLMs) scale to billions of parameters, leveraging expansive datasets, they exhibit enhanced linguistic capabilities and emergent abilities. However, this growth raises significant computational and privacy concerns. Our study addresses these issues by exploring the potential of MoE models, known for their computational efficiency, and the application of DP, a standard for privacy preservation. We present the first known attempt to train MoE models under the constraints of DP, addressing the unique challenges posed by their architecture and the complexities of DP integration. Our initial experimental studies demonstrate that MoE models can be effectively trained with DP, achieving performance that is competitive with their non-private counterparts. This initial study aims to provide valuable insights and ignite further research in the domain of privacy-preserving MoE models, softly laying the groundwork for prospective developments in this evolving field.

翻译：本立场论文研究了自然语言处理领域中混合专家（MoE）模型训练中集成差分隐私（DP）的技术。随着大语言模型（LLMs）扩展至数十亿参数并利用大规模数据集，它们展现出更强的语言能力和涌现特性。然而，这种规模增长引发了严重的计算与隐私问题。本研究通过探索以计算效率著称的MoE模型与作为隐私保护标准的DP技术的结合潜力来应对这些挑战。我们首次尝试在DP约束下训练MoE模型，解决了其架构带来的独特挑战以及DP集成的复杂性。初步实验研究表明，MoE模型可在差分隐私条件下有效训练，其性能与非隐私保护版本相当。本项前瞻性研究旨在为隐私保护型MoE领域提供重要见解并激发进一步探索，为这一快速发展领域的未来发展奠定基础。