This paper addresses the privacy and security concerns associated with deep neural language models, which serve as crucial components in various modern AI-based applications. These models are often used after being pre-trained and fine-tuned for specific tasks, with deployment on servers accessed through the internet. However, this introduces two fundamental risks: (a) the transmission of user inputs to the server via the network gives rise to interception vulnerabilities, and (b) privacy concerns emerge as organizations that deploy such models store user data with restricted context. To address this, we propose a novel method to adapt and fine-tune transformer-based language models on passkey-encrypted user-specific text. The original pre-trained language model first undergoes a quick adaptation (without any further pre-training) with a series of irreversible transformations applied to the tokenizer and token embeddings. This enables the model to perform inference on encrypted inputs while preventing reverse engineering of text from model parameters and intermediate outputs. After adaptation, models are fine-tuned on encrypted versions of existing training datasets. Experimental evaluation employing adapted versions of renowned models (e.g., BERT, RoBERTa) across established benchmark English and multilingual datasets for text classification and sequence labeling shows that encrypted models achieve performance parity with their original counterparts. This serves to safeguard performance, privacy, and security cohesively.

翻译：本文针对深度神经语言模型在现代AI应用中的隐私与安全问题展开研究。这类模型通常经过预训练和任务特定微调后部署于服务器，通过互联网提供服务。然而，这种部署模式存在两个根本性风险：（a）用户输入通过网络传输至服务器时存在拦截泄露隐患；（b）部署模型的组织在受限上下文存储用户数据引发隐私问题。为解决上述问题，我们提出一种新型方法，可对基于Transformer的语言模型进行适配与微调，使其能够处理经过密钥加密的用户特定文本。首先，原始预训练语言模型无需额外预训练，仅通过对分词器和词嵌入实施一系列不可逆变换即可快速完成适配，使模型能对加密输入执行推理，同时防止从模型参数和中间输出逆向还原原始文本。适配后，模型进一步在现有训练数据集的加密版本上进行微调。采用经适配的知名模型（如BERT、RoBERTa）在文本分类与序列标注的多个基准英文及多语言数据集上的实验评估表明，加密模型可实现与原模型相当的性能。这为统一保障性能、隐私与安全性提供了可行方案。