In this work, we carry out a data archaeology to infer books that are known to ChatGPT and GPT-4 using a name cloze membership inference query. We find that OpenAI models have memorized a wide collection of copyrighted materials, and that the degree of memorization is tied to the frequency with which passages of those books appear on the web. The ability of these models to memorize an unknown set of books complicates assessments of measurement validity for cultural analytics by contaminating test data; we show that models perform much better on memorized books than on non-memorized books for downstream tasks. We argue that this supports a case for open models whose training data is known.

翻译：在这项工作中，我们开展了一项数据考古学研究，通过名称完形填空式的成员推断查询，推断ChatGPT和GPT-4已知的书籍。研究发现，OpenAI模型记忆了广泛受版权保护的材料，且记忆程度与这些书籍片段在互联网上的出现频率相关。这些模型记忆未知书籍的能力，通过污染测试数据而影响了文化分析中测量有效性的评估：我们证明，在下游任务中，模型在已记忆书籍上的表现显著优于未记忆书籍。我们据此论证，这为训练数据公开的开放模型提供了支持依据。