We propose a privacy-preserving method for sharing text data by sharing noisy versions of their transformer embeddings. It has been shown that hidden representations learned by deep models can encode sensitive information from the input, making it possible for adversaries to recover the input data with considerable accuracy. This problem is exacerbated in transformer embeddings because they consist of multiple vectors, one per token. To mitigate this risk, we propose Nonparametric Variational Differential Privacy (NVDP), which ensures both useful data sharing and strong privacy protection. We take a differential privacy (DP) approach, integrating a nonparametric variational information bottleneck (NVIB) layer into the transformer architecture to inject noise into its multivector embeddings and thereby hide information, and measuring privacy protection with Rényi Divergence (RD) and its corresponding Bayesian Differential Privacy (BDP) guarantee. Training the NVIB layer calibrates the noise level according to the utility of the downstream task. We test NVDP on the General Language Understanding Evaluation (GLUE) benchmark and show that varying the noise level gives us a useful trade-off between privacy and accuracy. With lower noise levels, our model maintains high accuracy while offering strong privacy guarantees, effectively balancing privacy and utility.

翻译：我们提出了一种通过共享Transformer嵌入的噪声版本来实现文本数据隐私保护的共享方法。研究表明，深度学习模型学习到的隐藏表示能够编码输入数据中的敏感信息，使得攻击者能够以相当高的准确度恢复原始输入数据。这一问题在Transformer嵌入中尤为突出，因为其由多个向量构成（每个词符对应一个向量）。为降低此风险，我们提出了非参数变分差分隐私方法，该方法既能保证有效的数据共享，又能提供强大的隐私保护。我们采用差分隐私技术路径，将非参数变分信息瓶颈层集成到Transformer架构中，通过向多向量嵌入注入噪声来隐藏信息，并利用Rényi散度及其对应的贝叶斯差分隐私保证来衡量隐私保护强度。通过训练NVIB层，可以根据下游任务的实际效用校准噪声水平。我们在通用语言理解评估基准上测试了NVDP方法，结果表明通过调节噪声水平可以实现隐私保护与模型精度的有效权衡。在较低噪声水平下，我们的模型在提供强隐私保障的同时仍能保持较高精度，实现了隐私保护与实用性的有效平衡。