The performance of Large Language Models (LLMs) is determined by their training data. Despite the proliferation of open-weight LLMs, access to LLM training data has remained limited. Even for fully open LLMs, the scale of the data makes it all but inscrutable to the general scientific community, despite potentially containing critical data scraped from the internet. In this paper, we present the full-text indexing pipeline for the Apertus LLM training data. Leveraging Elasticsearch parallel indices and the Alps infrastructure, a state-of-the-art, highly energy-efficient arm64 supercluster, we were able to index 8.6T tokens out of 15.2T used to train the Apertus LLM family, creating both a critical LLM safety tool and effectively an offline, curated, open web search engine. Our contribution is threefold. First, we demonstrate that Elasticsearch can be successfully ported onto next-generation arm64-based infrastructure. Second, we demonstrate that full-text indexing at the scale of modern LLM training datasets and the entire open web is feasible and accessible. Finally, we demonstrate that such indices can be used to ensure previously inaccessible jailbreak-agnostic LLM safety. We hope that our findings will be useful to other teams attempting large-scale data indexing and facilitate the general transition towards greener computation.

翻译：大型语言模型（LLM）的性能由其训练数据决定。尽管开源权重的LLM不断涌现，但LLM训练数据的获取仍然受限。即使对于完全开源的LLM，其数据规模之大也使得科学界难以深入解析，尽管其中可能包含从互联网抓取的关键数据。本文介绍了针对Apertus LLM训练数据的全文索引构建流程。通过结合Elasticsearch并行索引技术与Alps基础设施——一个基于arm64架构的尖端高能效超级计算集群，我们成功对Apertus LLM系列训练所用的15.2万亿token中的8.6万亿进行了索引，既构建了关键的LLM安全工具，也创建了一个离线、经过筛选的开放式网络搜索引擎。我们的贡献主要体现在三个方面：首先，验证了Elasticsearch能够成功移植到新一代arm64基础设施；其次，证明了在现代LLM训练数据集及整个开放网络规模下实施全文索引具有可行性与可操作性；最后，展示了此类索引可用于实现以往难以达成的、对越狱攻击不敏感的LLM安全保障。我们期望本研究能为其他团队的大规模数据索引工作提供参考，并推动计算生态向绿色化转型。