Pre-trained large language models, such as GPT\nobreakdash-2 and BERT, are often fine-tuned to achieve state-of-the-art performance on a downstream task. One natural example is the ``Smart Reply'' application where a pre-trained model is tuned to provide suggested responses for a given query message. Since the tuning data is often sensitive data such as emails or chat transcripts, it is important to understand and mitigate the risk that the model leaks its tuning data. We investigate potential information leakage vulnerabilities in a typical Smart Reply pipeline. We consider a realistic setting where the adversary can only interact with the underlying model through a front-end interface that constrains what types of queries can be sent to the model. Previous attacks do not work in these settings, but require the ability to send unconstrained queries directly to the model. Even when there are no constraints on the queries, previous attacks typically require thousands, or even millions, of queries to extract useful information, while our attacks can extract sensitive data in just a handful of queries. We introduce a new type of active extraction attack that exploits canonical patterns in text containing sensitive data. We show experimentally that it is possible for an adversary to extract sensitive user information present in the training data, even in realistic settings where all interactions with the model must go through a front-end that limits the types of queries. We explore potential mitigation strategies and demonstrate empirically how differential privacy appears to be a reasonably effective defense mechanism to such pattern extraction attacks.

翻译：预训练的大型语言模型（如GPT-2和BERT）通常经过微调以在特定下游任务中实现最优性能。一个典型的例子是“智能回复”应用，其中预训练模型被调优以针对给定查询消息提供建议回复。由于调优数据通常是敏感数据（如电子邮件或聊天记录），理解并降低模型泄露其调优数据的风险至关重要。我们研究了典型智能回复流程中潜在的信息泄露漏洞。考虑一种现实场景：攻击者只能通过一个限制查询类型的前端接口与底层模型交互。现有攻击方法在此类设置中无法生效，因为它们要求直接向模型发送无约束查询。即使查询不受限制，现有攻击通常需要数千甚至数百万次查询才能提取有用信息，而我们的攻击仅需少量查询即可提取敏感数据。我们提出一种新型主动抽取攻击方法，利用包含敏感数据的文本中的规范模式。实验表明，即使在所有模型交互必须通过限制查询类型的前端接口进行的现实场景中，攻击者仍能提取训练数据中的敏感用户信息。我们探索了潜在的防御策略，并通过实验证明差分隐私对此类模式抽取攻击似乎是相当有效的防御机制。