Detecting whether copyright holders' works were used in LLM pretraining is poised to be an important problem. This work proposes using data watermarks to enable principled detection with only black-box model access, provided that the rightholder contributed multiple training documents and watermarked them before public release. By applying a randomly sampled data watermark, detection can be framed as hypothesis testing, which provides guarantees on the false detection rate. We study two watermarks: one that inserts random sequences, and another that randomly substitutes characters with Unicode lookalikes. We first show how three aspects of watermark design -- watermark length, number of duplications, and interference -- affect the power of the hypothesis test. Next, we study how a watermark's detection strength changes under model and dataset scaling: while increasing the dataset size decreases the strength of the watermark, watermarks remain strong if the model size also increases. Finally, we view SHA hashes as natural watermarks and show that we can robustly detect hashes from BLOOM-176B's training data, as long as they occurred at least 90 times. Together, our results point towards a promising future for data watermarks in real world use.

翻译：检测版权持有者的作品是否被用于大语言模型（LLM）预训练，正成为一个重要问题。本文提出利用数据水印实现原则性检测，仅需黑盒模型访问权限，前提是权利人在公开发布前已贡献多个训练文档并为其添加水印。通过应用随机采样的数据水印，可将检测问题转化为假设检验，从而为错误检测率提供理论保障。我们研究了两种水印方法：一种插入随机序列，另一种用Unicode同形字符随机替换字符。首先揭示水印设计的三个维度——水印长度、重复次数与干扰——如何影响假设检验的统计功效。其次，研究模型规模和数据集规模扩展对水印检测强度的影响：虽然增大数据集会削弱水印强度，但若同时扩大模型规模，水印仍能保持较强检测能力。最后，将SHA哈希视为天然水印，证明当BLOOM-176B的训练数据中某哈希出现至少90次时，可稳健检测该哈希的存在。综合结果表明，数据水印在实际应用中具有广阔前景。