In very recent years more attention has been placed on probing the role of pre-training data in Large Language Models (LLMs) downstream behaviour. Despite the importance, there is no public tool that supports such analysis of pre-training corpora at large scale. To help research in this space, we launch Koala, a searchable index over large pre-training corpora using compressed suffix arrays with highly efficient compression rate and search support. In its first release we index the public proportion of OPT 175B pre-training data. Koala provides a framework to do forensic analysis on the current and future benchmarks as well as to assess the degree of memorization in the output from the LLMs. Koala is available for public use at https://koala-index.erc.monash.edu/.

翻译：近年来，学界愈发关注预训练数据在大语言模型（LLMs）下游行为中的作用。尽管这一课题意义重大，但目前尚无支持大规模预训练语料库此类分析的公开工具。为助力相关研究，我们推出Koala——一个基于压缩后缀数组构建的预训练语料库可搜索索引，兼具高压缩率与高效搜索能力。在其首个版本中，我们索引了OPT 175B预训练数据的公开部分。Koala提供了一套分析框架，可用于对现有及未来基准测试进行取证分析，并评估LLMs输出中的记忆化程度。Koala现已面向公众开放，访问地址为：https://koala-index.erc.monash.edu/。