Large Language Models (LLMs), such as BERT and GPT-based models like ChatGPT, have recently demonstrated their impressive capacity for learning language representations, yielding significant benefits for various downstream Natural Language Processing (NLP) tasks. However, the immense data requirements of these large models have incited substantial concerns regarding copyright protection and data privacy. In an attempt to address these issues, particularly the unauthorized use of private data in LLMs, we introduce a novel watermarking technique via a backdoor-based membership inference approach, i.e., TextMarker, which can safeguard diverse forms of private information embedded in the training text data in LLMs. Specifically, TextMarker is a new membership inference framework that can eliminate the necessity for additional proxy data and surrogate model training, which are common in traditional membership inference techniques, thereby rendering our proposal significantly more practical and applicable.

翻译：大型语言模型（LLMs），如BERT及基于GPT的模型（例如ChatGPT），近期在语言表征学习方面展现出令人印象深刻的能力，为各类下游自然语言处理（NLP）任务带来了显著收益。然而，这些大模型对海量数据的需求引发了关于版权保护与数据隐私的严重关切。为解决这些问题，特别是针对LLMs中未经授权使用私有数据的情况，我们提出了一种创新的水印技术，该技术基于后门驱动的成员推断方法，即TextMarker，能够保护LLMs训练文本数据中嵌入的多种形式的私有信息。具体而言，TextMarker是一种新颖的成员推断框架，它消除了传统成员推断技术中常见的对额外代理数据及替代模型训练的依赖，从而使我们的方案更具实用性与适用性。