The drastic increase in language models' parameters has led to a new trend of deploying models in cloud servers, raising growing concerns about private inference for Transformer-based models. Existing two-party privacy-preserving techniques, however, only take into account natural language understanding (NLU) scenarios. Private inference in natural language generation (NLG), crucial for applications like translation and code completion, remains underexplored.In addition, previous privacy-preserving techniques suffer from convergence issues during model training and exhibit poor inference speed when used with NLG models due to the neglect of time-consuming operations in auto-regressive generations. To address these issues, we propose a fast private text generation framework for Transformer-based language models, namely MERGE.MERGE reuses the output hidden state as the word embedding to bypass the embedding computation and reorganize the linear operations in the Transformer module to accelerate the forward procedure. Extensive experiments show that MERGE achieves a 26.5x speedup to the vanilla encrypted model under the sequence length 512, and reduces 80\% communication cost, with an up to 10x speedup to state-of-the-art approximated models.

翻译：语言模型参数量急剧增长，导致模型部署向云端服务器迁移的新趋势日益显著，由此引发基于Transformer模型的隐私推理问题广受关注。现有两方隐私保护技术仅适用于自然语言理解场景，而自然语言生成中的隐私推理——这一对翻译、代码补全等应用至关重要的领域——仍存在研究空白。此外，由于忽视自回归生成中的高耗时运算，既有隐私保护技术在模型训练中面临收敛性问题，应用于自然语言生成模型时推理速度极差。针对上述问题，我们提出面向Transformer语言模型的快速私有文本生成框架MERGE。该框架通过复用输出隐状态作为词嵌入以规避嵌入计算，并重组Transformer模块中的线性运算来加速前向过程。大量实验表明，当序列长度为512时，MERGE可实现相比原始加密模型26.5倍的加速比，降低80%通信开销，相比当前最优近似模型获得最高10倍加速。