Suffix arrays power deduplication, contamination testing, and training data attribution over the corpora used to train large language models, and must be rebuilt for every new corpus or tokenizer. However, constructing them at scale is slow. infini-gram's indexing pipeline takes about two days to process 1.4 trillion tokens on its authors' 128-core CPU server. GPU-accelerated constructors are an order of magnitude faster, but their working sets must fit in GPU memory. Partitioning the corpus only moves this limit to host memory. Sorting the suffixes of one partition needs the ranks of suffixes in other partitions, which existing designs keep in a corpus-wide rank table that must stay in memory. We present Cascade, a GPU-accelerated out-of-core suffix-array constructor. Its partitions each own a fixed, contiguous range of the final suffix array, so the GPU sorts them independently and never merges them. Cascade then replaces the rank table with streamed rank exchange. Ranks are sent directly to the partitions that need them, and an invariant makes partition-local ranks sufficient for every comparison. With no rank table, intermediate data spills across GPU memory, host memory, and SSD as the corpus grows, and host memory only needs to hold the text. On seven corpora of 8-238 billion tokens, Cascade builds suffix arrays on a single GPU server 3.0-8.2x faster end to end than infini-gram's pipeline, and its construction is 6.3-11.4x faster than libsais's. It indexes the 198-billion-token C4 corpus and the 238-billion-token Pile in 23 and 37 minutes end to end, both of which exceed libsais's host-memory capacity on the same server. A batched query engine over the index reproduces infini-gram's counts at median speedups of 10.8-32.6x, with the index in GPU memory, host memory, or on SSD.

翻译：暂无翻译