With the growing use of Transformer models hosted on cloud platforms to offer inference services, privacy concerns are escalating, especially concerning sensitive data like investment plans and bank account details. Secure Multi-Party Computing (SMPC) emerges as a promising solution to protect the privacy of inference data and model parameters. However, the application of SMPC in Privacy-Preserving Inference (PPI) for Transformer models often leads to considerable slowdowns or declines in performance. This is largely due to the multitude of nonlinear operations in the Transformer architecture, which are not well-suited to SMPC and difficult to circumvent or optimize effectively. To address this concern, we introduce a comprehensive PPI framework called SecFormer to achieve fast and accurate PPI for Transformer models. We successfully eliminate the high-cost exponential and maximum operations in PPI without sacrificing model performance and develop a suite of efficient SMPC protocols by employing suitable numerical computation methods to boost other complex nonlinear functions in PPI, including GeLU, LayerNorm, and a redesigned Softmax. Our extensive experiments reveal that SecFormer outperforms MPCFormer in performance, showing improvements of $3.4\%$ and $24.7\%$ for BERT$_{\text{BASE}}$ and BERT$_{\text{LARGE}}$, respectively. In terms of efficiency, SecFormer is 3.57 and 3.58 times faster than PUMA for BERT$_{\text{BASE}}$ and BERT$_{\text{LARGE}}$, demonstrating its effectiveness and speed.

翻译：随着托管在云平台上提供推理服务的Transformer模型日益普及，隐私问题不断加剧，尤其是涉及投资计划、银行账户详情等敏感数据时。安全多方计算（SMPC）作为一种保护推理数据与模型参数隐私的有效方案受到广泛关注。然而，将SMPC应用于Transformer模型的隐私保护推理（PPI）常导致显著的性能下降或速度损失。这主要源于Transformer架构中存在大量非线性运算，这些运算与SMPC兼容性较差且难以有效规避或优化。为解决该问题，本文提出名为SecFormer的完整PPI框架，旨在实现Transformer模型的高效精准隐私保护推理。我们在不损失模型性能的前提下，成功消除了PPI中代价高昂的指数运算与最大值运算，并通过采用合适的数值计算方法优化了PPI中其他复杂非线性函数（包括GeLU、LayerNorm及重新设计的Softmax），开发出一系列高效SMPC协议。大量实验表明，SecFormer在性能上优于MPCFormer：在BERT$_{\text{BASE}}$与BERT$_{\text{LARGE}}$模型上分别实现了$3.4\%$和$24.7\%$的性能提升。在效率方面，SecFormer对BERT$_{\text{BASE}}$和BERT$_{\text{LARGE}}$的推理速度分别达到PUMA的3.57倍与3.58倍，充分验证了其高效性与实用性。