With the growing use of large language models hosted on cloud platforms to offer inference services, privacy concerns are escalating, especially concerning sensitive data like investment plans and bank account details. Secure Multi-Party Computing (SMPC) emerges as a promising solution to protect the privacy of inference data and model parameters. However, the application of SMPC in Privacy-Preserving Inference (PPI) for large language models, particularly those based on the Transformer architecture, often leads to considerable slowdowns or declines in performance. This is largely due to the multitude of nonlinear operations in the Transformer architecture, which are not well-suited to SMPC and difficult to circumvent or optimize effectively. To address this concern, we introduce an advanced optimization framework called SecFormer, to achieve fast and accurate PPI for Transformer models. By implementing model design optimization, we successfully eliminate the high-cost exponential and maximum operations in PPI without sacrificing model performance. Additionally, we have developed a suite of efficient SMPC protocols that utilize segmented polynomials, Fourier series and Goldschmidt's method to handle other complex nonlinear functions within PPI, such as GeLU, LayerNorm, and Softmax. Our extensive experiments reveal that SecFormer outperforms MPCFormer in performance, showing improvements of $5.6\%$ and $24.2\%$ for BERT$_{\text{BASE}}$ and BERT$_{\text{LARGE}}$, respectively. In terms of efficiency, SecFormer is 3.56 and 3.58 times faster than Puma for BERT$_{\text{BASE}}$ and BERT$_{\text{LARGE}}$, demonstrating its effectiveness and speed.

翻译：随着大型语言模型被部署在云平台上提供推理服务的日益普及，隐私问题日益凸显，尤其是涉及投资计划和银行账户详情等敏感数据时。安全多方计算（SMPC）成为保护推理数据和模型参数隐私的一种有前景的解决方案。然而，将SMPC应用于大型语言模型（特别是基于Transformer架构的模型）的隐私保护推理（PPI）中，常常会导致推理速度显著下降或性能降低。这主要是由于Transformer架构中包含大量非线性运算，这些运算对SMPC兼容性不佳，且难以有效规避或优化。为解决这一问题，我们提出了一种名为SecFormer的高级优化框架，旨在实现对Transformer模型的快速且精准的PPI。通过实施模型设计优化，我们成功消除了PPI中的高成本指数运算和最大值运算，且不影响模型性能。此外，我们开发了一套高效的SMPC协议，利用分段多项式、傅里叶级数和Goldschmidt方法来处理PPI中的其他复杂非线性函数，如GeLU、LayerNorm和Softmax。广泛实验表明，SecFormer在性能上优于MPCFormer，对于BERT$_{\text{BASE}}$和BERT$_{\text{LARGE}}$模型，分别提升了$5.6\%$和$24.2\%$。在效率方面，对于BERT$_{\text{BASE}}$和BERT$_{\text{LARGE}}$，SecFormer的推理速度分别是Puma的3.56倍和3.58倍，充分展现了其高效性与优越性。