With the growing use of large language models hosted on cloud platforms to offer inference services, privacy concerns are escalating, especially concerning sensitive data like investment plans and bank account details. Secure Multi-Party Computing (SMPC) emerges as a promising solution to protect the privacy of inference data and model parameters. However, the application of SMPC in Privacy-Preserving Inference (PPI) for large language models, particularly those based on the Transformer architecture, often leads to considerable slowdowns or declines in performance. This is largely due to the multitude of nonlinear operations in the Transformer architecture, which are not well-suited to SMPC and are difficult to circumvent or optimize effectively. To address this concern, we introduce an advanced optimization framework called SecFormer, designed to strike an optimal balance between performance and efficiency in PPI for Transformer models. By implementing knowledge distillation techniques, we successfully eliminate the high-cost exponential and maximum operations in PPI without sacrificing model performance. Additionally, we have developed a suite of efficient SMPC protocols that utilize segmented polynomials and Goldschmidt's method to handle other complex nonlinear functions within PPI, such as GeLU, LayerNorm, and Softmax. Our extensive experiments reveal that SecFormer outperforms MPCFormer in performance, showing improvements of $5.6\%$ and $24.2\%$ for BERT$_{\text{BASE}}$ and BERT$_{\text{LARGE}}$, respectively. In terms of efficiency, SecFormer is 3.4 and 3.2 times faster than Puma, demonstrating its effectiveness and speed.

翻译：随着云端平台托管大型语言模型提供推理服务的日益普及，隐私问题愈发严峻，尤其是涉及投资计划和银行账户详情等敏感数据时。安全多方计算（SMPC）成为保护推理数据和模型参数隐私的有前景解决方案。然而，将SMPC应用于大型语言模型（特别是基于Transformer架构的模型）的隐私保护推理（PPI），往往会导致显著的速度降低或性能下降。这主要源于Transformer架构中存在大量非线性运算，这些运算难以适配SMPC，且难以有效规避或优化。为解决这一问题，我们提出了一种名为SecFormer的先进优化框架，旨在为Transformer模型的PPI实现性能与效率的最优平衡。通过应用知识蒸馏技术，我们成功消除了PPI中的高成本指数运算和最大值运算，且不牺牲模型性能。此外，我们开发了一套高效的SMPC协议，利用分段多项式与Goldschmidt方法处理PPI中其他复杂非线性函数（如GeLU、LayerNorm和Softmax）。大量实验表明，SecFormer在性能上优于MPCFormer，在BERT$_{\text{BASE}}$和BERT$_{\text{LARGE}}$上分别提升$5.6\%$和$24.2\%$；在效率上，SecFormer比Puma快3.4倍和3.2倍，充分证明了其高效性与速度优势。