We propose a privacy-preserving method for sharing text data by sharing noisy versions of their transformer embeddings. It has been shown that hidden representations learned by deep models can encode sensitive information from the input, making it possible for adversaries to recover the input data with considerable accuracy. This problem is exacerbated in transformer embeddings because they consist of multiple vectors, one per token. To mitigate this risk, we propose Nonparametric Variational Differential Privacy (NVDP), which ensures both useful data sharing and strong privacy protection. We take a differential privacy approach, integrating a Nonparametric Variational Information Bottleneck (NVIB) layer into the transformer architecture to inject noise into its multi-vector embeddings and thereby hide information, and measuring privacy protection with Rényi divergence and its corresponding Bayesian Differential Privacy (BDP) guarantee. Training the NVIB layer calibrates the noise level according to utility. We test NVDP on the GLUE benchmark and show that varying the noise level gives us a useful tradeoff between privacy and accuracy. With lower noise levels, our model maintains high accuracy while offering strong privacy guarantees, effectively balancing privacy and utility.

翻译：我们提出了一种通过共享Transformer嵌入的噪声版本来实现文本数据隐私保护的共享方法。研究表明，深度模型学习到的隐藏表示能够编码输入中的敏感信息，使得攻击者能够以相当高的准确度恢复原始输入数据。这一问题在Transformer嵌入中尤为突出，因为其由多个向量构成（每个词符对应一个向量）。为降低此风险，我们提出了非参数变分差分隐私方法，该方法既能保证有效的数据共享，又能提供强大的隐私保护。我们采用差分隐私框架，将非参数变分信息瓶颈层集成到Transformer架构中，通过向多向量嵌入注入噪声来隐藏信息，并利用Rényi散度及其对应的贝叶斯差分隐私保证来度量隐私保护强度。训练NVIB层可根据效用需求校准噪声水平。我们在GLUE基准测试上验证了NVDP方法，结果表明通过调节噪声水平可在隐私保护与模型精度之间实现有效权衡。在较低噪声水平下，我们的模型在提供强隐私保证的同时仍能保持较高精度，实现了隐私保护与数据效用的有效平衡。