LLM-powered chatbots are becoming widely adopted in applications such as healthcare, personal assistants, industry hiring decisions, etc. In many of these cases, chatbots are fed sensitive, personal information in their prompts, as samples for in-context learning, retrieved records from a database, or as part of the conversation. The information provided in the prompt could directly appear in the output, which might have privacy ramifications if there is sensitive information there. As such, in this paper, we aim to understand the input copying and regurgitation capabilities of these models during inference and how they can be directly instructed to limit this copying by complying with regulations such as HIPAA and GDPR, based on their internal knowledge of them. More specifically, we find that when ChatGPT is prompted to summarize cover letters of a 100 candidates, it would retain personally identifiable information (PII) verbatim in 57.4% of cases, and we find this retention to be non-uniform between different subgroups of people, based on attributes such as gender identity. We then probe ChatGPT's perception of privacy-related policies and privatization mechanisms by directly instructing it to provide compliant outputs and observe a significant omission of PII from output.

翻译：基于大型语言模型的聊天机器人正广泛应用于医疗、个人助理、行业招聘决策等领域。在许多场景中，提示词中会包含敏感个人信息——这些信息可能来源于上下文学习的样本、数据库检索记录或对话内容。提示词中的信息可能直接出现在输出中，若包含敏感数据则可能引发隐私风险。为此，本文旨在探究这些模型在推理阶段对输入内容的复制与复现能力，以及如何通过直接指令约束其根据HIPAA和GDPR等法规的内部知识限制这种复制行为。具体而言，我们发现当ChatGPT被要求总结100份求职信时，在57.4%的案例中会原样保留个人身份信息，且这种保留现象在不同人群（如基于性别认同的群体）间呈现非均匀分布。通过直接指令要求其输出符合隐私政策时，我们进一步探测了ChatGPT对隐私相关法规及脱敏机制的理解，观察到其输出中个人身份信息被显著移除的现象。