As the deployment of pre-trained language models (PLMs) expands, pressing security concerns have arisen regarding the potential for malicious extraction of training data, posing a threat to data privacy. This study is the first to provide a comprehensive survey of training data extraction from PLMs. Our review covers more than 100 key papers in fields such as natural language processing and security. First, preliminary knowledge is recapped and a taxonomy of various definitions of memorization is presented. The approaches for attack and defense are then systemized. Furthermore, the empirical findings of several quantitative studies are highlighted. Finally, future research directions based on this review are suggested.

翻译：随着预训练语言模型（PLMs）部署规模的扩大，关于其训练数据可能被恶意提取的紧迫安全问题日益凸显，这对数据隐私构成了威胁。本研究首次对从PLMs中提取训练数据进行了全面综述。我们综述了涵盖自然语言处理和网络安全等领域的一百余篇关键论文。首先，回顾了预备知识，并提出了关于记忆化多种定义的分类体系。随后，系统化了攻击与防御方法。此外，还重点介绍了多项定量研究的实证结果。最后，基于本综述提出了未来研究方向。