In this paper, we explore the idea of training large language models (LLMs) over highly compressed text. While standard subword tokenizers compress text by a small factor, neural text compressors can achieve much higher rates of compression. If it were possible to train LLMs directly over neurally compressed text, this would confer advantages in training and serving efficiency, as well as easier handling of long text spans. The main obstacle to this goal is that strong compression tends to produce opaque outputs that are not well-suited for learning. In particular, we find that text na\"ively compressed via Arithmetic Coding is not readily learnable by LLMs. To overcome this, we propose Equal-Info Windows, a novel compression technique whereby text is segmented into blocks that each compress to the same bit length. Using this method, we demonstrate effective learning over neurally compressed text that improves with scale, and outperforms byte-level baselines by a wide margin on perplexity and inference speed benchmarks. While our method delivers worse perplexity than subword tokenizers for models trained with the same parameter count, it has the benefit of shorter sequence lengths. Shorter sequence lengths require fewer autoregressive generation steps, and reduce latency. Finally, we provide extensive analysis of the properties that contribute to learnability, and offer concrete suggestions for how to further improve the performance of high-compression tokenizers.

翻译：本文探讨了在高度压缩文本上训练大型语言模型（LLMs）的设想。虽然标准子词分词器仅能以较小的倍数压缩文本，但神经文本压缩器可实现更高的压缩率。若能直接在神经压缩文本上训练LLMs，将带来训练与推理效率的提升，并便于处理长文本片段。实现该目标的主要障碍在于：强压缩往往产生不透明的输出，不利于学习。特别地，我们发现经算术编码朴素压缩后的文本难以被LLMs习得。为此，我们提出“等信息窗口”（Equal-Info Windows）这一新型压缩技术，将文本分割成多个等比特长度的压缩块。通过该方法，我们证明了在神经压缩文本上的有效学习能随模型规模提升而改进，且在困惑度与推理速度基准上大幅超越基于字节的基线模型。尽管对相同参数量的模型，我们的方法产生的困惑度劣于子词分词器，但其优势在于更短的序列长度——较短的序列需更少的自回归生成步骤并降低延迟。最后，我们深入分析了影响可学习性的属性特征，并为进一步提升高压缩分词器性能提供了具体建议。