Privacy preserving deep learning is an emerging field in machine learning that aims to mitigate the privacy risks in the use of deep neural networks. One such risk is training data extraction from language models that have been trained on datasets, which contain personal and privacy sensitive information. In our study, we investigate the extent of named entity memorization in fine-tuned BERT models. We use single-label text classification as representative downstream task and employ three different fine-tuning setups in our experiments, including one with Differentially Privacy (DP). We create a large number of text samples from the fine-tuned BERT models utilizing a custom sequential sampling strategy with two prompting strategies. We search in these samples for named entities and check if they are also present in the fine-tuning datasets. We experiment with two benchmark datasets in the domains of emails and blogs. We show that the application of DP has a detrimental effect on the text generation capabilities of BERT. Furthermore, we show that a fine-tuned BERT does not generate more named entities specific to the fine-tuning dataset than a BERT model that is pre-trained only. This suggests that BERT is unlikely to emit personal or privacy sensitive named entities. Overall, our results are important to understand to what extent BERT-based services are prone to training data extraction attacks.

翻译：隐私保护深度学习是机器学习领域的一个新兴方向，旨在缓解深度神经网络使用中的隐私风险。其中一项风险是从基于包含个人和隐私敏感信息的数据集训练的语言模型中提取训练数据。本研究探究了微调BERT模型中的命名实体记忆化程度。我们以单标签文本分类作为代表性下游任务，在实验中采用三种不同的微调设置（包括差分隐私设置）。利用自定义序贯采样策略结合两种提示策略，我们从微调BERT模型中生成了大量文本样本，并在这些样本中搜索命名实体，验证其是否存在于微调数据集中。我们使用电子邮件和博客领域的两个基准数据集进行实验。结果表明，应用差分隐私对BERT的文本生成能力产生不利影响；此外，微调BERT生成的微调数据集特有命名实体数量并不多于仅经过预训练的BERT模型——这表明BERT不太可能泄露个人或隐私敏感的命名实体。总体而言，我们的研究结果对理解基于BERT的服务在多大程度上容易遭受训练数据提取攻击具有重要意义。