Understanding how language-model outputs relate to the pretraining corpus is central to studying model behavior. Most training data attribution (TDA) methods ask which training examples causally influence a given output, often using leave-one-out tests. We invert the question: which outputs cannot be attributed to any pretraining example? We introduce un-attributability as an operational measure of semantic novelty: an output is novel if the pretraining corpus contains no semantically similar context. We approximate this with a simple two-stage retrieval pipeline: index the corpus with lightweight GIST embeddings, retrieve the top-n candidates, then rerank with ColBERTv2. If the nearest corpus item is less attributable than a human-generated text reference, we consider the output of the model as novel. We evaluate on SmolLM and SmolLM2 and report three findings: (1) models draw on pretraining data across much longer spans than previously reported; (2) some domains systematically promote or suppress novelty; and (3) instruction tuning not only alters style but also increases novelty. Reframing novelty assessment around un-attributability enables efficient analysis at pretraining scale. We release ~20 TB of corpus chunks and index artifacts to support replication and large-scale extension of our analysis at https://huggingface.co/datasets/stai-tuebingen/faiss-smollm

翻译：理解语言模型输出与预训练语料之间的关系是研究模型行为的核心。多数训练数据归因（TDA）方法通过留一测试探究哪些训练样本对特定输出具有因果影响。我们反转了这一问题：哪些输出无法归因于任何预训练样本？我们引入不可归因性作为语义新颖性的操作性度量：若预训练语料中不存在语义相似的上下文，则视该输出具有新颖性。我们通过一个简单的两阶段检索流程进行近似：首先使用轻量级GIST嵌入对语料建立索引并检索前n个候选，随后通过ColBERTv2进行重排序。若最近邻语料项的归因度低于人工生成的文本参照，则判定模型输出具有新颖性。我们在SmolLM与SmolLM2模型上开展评估并报告三项发现：（1）模型依赖的预训练数据跨度远超以往报道；（2）某些领域会系统性促进或抑制新颖性；（3）指令微调不仅改变风格，同时提升新颖性。围绕不可归因性重构新颖性评估框架，可实现预训练规模下的高效分析。我们在https://huggingface.co/datasets/stai-tuebingen/faiss-smollm 发布了约20 TB的语料分块与索引数据，以支持本研究的复现及大规模扩展分析。