Large Language Models (LLMs), trained on massive corpora with billions of parameters, show unprecedented performance in various fields. Though surprised by their excellent performances, researchers also noticed some special behaviors of those LLMs. One of those behaviors is memorization, in which LLMs can generate the same content used to train them. Though previous research has discussed memorization, the memorization of LLMs still lacks explanation, especially the cause of memorization and the dynamics of generating them. In this research, we comprehensively discussed memorization from various perspectives and extended the discussion scope to not only just the memorized content but also less and unmemorized content. Through various studies, we found that: (1) Through experiments, we revealed the relation of memorization between model size, continuation size, and context size. Further, we showed how unmemorized sentences transition to memorized sentences. (2) Through embedding analysis, we showed the distribution and decoding dynamics across model size in embedding space for sentences with different memorization scores. The n-gram statistics analysis presents d (3) An analysis over n-gram and entropy decoding dynamics discovered a boundary effect when the model starts to generate memorized sentences or unmemorized sentences. (4)We trained a Transformer model to predict the memorization of different models, showing that it is possible to predict memorizations by context.

翻译：大型语言模型（LLMs）在拥有数十亿参数的海量语料上训练后，在各个领域展现出前所未有的性能。尽管其卓越表现令人惊叹，研究人员也注意到这些LLM的一些特殊行为，其中之一便是记忆化现象——LLMs能够生成与训练数据完全相同的内容。虽然已有研究探讨过记忆化，但LLM的记忆化机制仍缺乏合理解释，尤其是其成因与生成动态过程。在本研究中，我们从多个视角全面探讨记忆化现象，并将讨论范围从记忆化内容扩展至低记忆化和非记忆化内容。通过多项研究我们发现：(1) 实验揭示了记忆化与模型规模、续写规模及上下文规模之间的关系，并展示了非记忆化句子向记忆化句子的转变过程；(2) 通过嵌入分析，我们展示了不同记忆化评分句子的嵌入空间分布及跨模型规模的解码动态，而n-gram统计分析则呈现了；(3) 对n-gram与熵解码动态的分析发现，当模型开始生成记忆化或非记忆化句子时存在边界效应；(4) 我们训练了一个Transformer模型来预测不同模型的记忆化程度，表明通过上下文预测记忆化是可行的。