It is increasingly important to enable privacy-preserving inference for cloud services based on Transformers. Post-quantum cryptographic techniques, e.g., fully homomorphic encryption (FHE), and multi-party computation (MPC), are popular methods to support private Transformer inference. However, existing works still suffer from prohibitively computational and communicational overhead. In this work, we present, Primer, to enable a fast and accurate Transformer over encrypted data for natural language processing tasks. In particular, Primer is constructed by a hybrid cryptographic protocol optimized for attention-based Transformer models, as well as techniques including computation merge and tokens-first ciphertext packing. Comprehensive experiments on encrypted language modeling show that Primer achieves state-of-the-art accuracy and reduces the inference latency by 90.6% ~ 97.5% over previous methods.

翻译：在基于Transformer的云服务中实现隐私保护推理正变得日益重要。后量子密码技术，例如全同态加密（FHE）与多方计算（MPC），是支持隐私Transformer推理的常用方法。然而，现有方案仍面临计算与通信开销过高的难题。本文提出Primer系统，通过构建针对注意力机制Transformer模型的混合密码协议，结合计算合并与令牌优先密文打包技术，实现对加密数据上自然语言处理任务的快速精准推理。在加密语言建模任务上的综合实验表明，Primer在达到现有最优准确率的同时，将推理延迟较此前方法降低了90.6%至97.5%。