Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. However, existing approaches operate at the single-document level, rewriting individual web pages in isolation. This confines synthesized examples to intra-document knowledge, missing cross-document relationships and leaving facts with limited associative context. We propose WRAP++ (Web discoveRy Amplified Pretraining), which amplifies the associative context of factual knowledge by discovering cross-document relationships from web hyperlinks and synthesizing joint QA over each discovered document pair. Concretely, WRAP++ discovers high-confidence relational motifs including dual-links and co-mentions, and synthesizes QA that requires reasoning across both documents. This produces relational knowledge absent from either source document alone, creating diverse entry points to the same facts. Because the number of valid entity pairs grows combinatorially, this discovery-driven synthesis also amplifies data scale far beyond single-document rewriting. Instantiating WRAP++ on Wikipedia, we amplify ~8.4B tokens of raw text into 80B tokens of cross-document QA data. On SimpleQA, OLMo-based models at both 7B and 32B scales trained with WRAP++ substantially outperform single-document approaches and exhibit sustained scaling gains, underscoring the advantage of cross-document knowledge discovery and amplification.

翻译：合成数据改写已成为提升大语言模型预训练过程中知识获取能力的有力技术。然而，现有方法仅在单文档层面操作，孤立地重写单个网页。这限制了合成示例仅涵盖文档内部知识，忽略了跨文档关系，并使得事实缺乏丰富的关联上下文。我们提出WRAP++（网络发现增强预训练），该方法通过从网页超链接中发现跨文档关系，并对每个发现的文档对合成联合问答，从而增强事实性知识的关联上下文。具体而言，WRAP++发现高置信度的关系模式，包括双向链接和共提及，并合成需要跨两个文档进行推理的问答。这产生了任一源文档本身所不具备的关系性知识，为同一事实创造了多样化的切入点。由于有效实体对的数量呈组合式增长，这种基于发现的合成方式也显著扩大了数据规模，远超单文档重写。在维基百科上实例化WRAP++后，我们将约84亿token的原始文本扩充为800亿token的跨文档问答数据。在SimpleQA上，基于OLMo的模型在7B和32B两种规模下，使用WRAP++训练后均显著优于单文档方法，并展现出持续的规模增益，凸显了跨文档知识发现与增强的优势。