Modern language models are trained almost exclusively on token sequences produced by a fixed tokenizer, an external lossless compressor often over UTF-8 byte sequences, thereby coupling the model to that compressor. This work introduces proxy compression, an alternative training scheme that preserves the efficiency benefits of compressed inputs while providing an end-to-end, raw-byte interface at inference time. During training, one language model is jointly trained on raw byte sequences and compressed views generated by external compressors; through the process, the model learns to internally align compressed sequences and raw bytes. This alignment enables strong transfer between the two formats, even when training predominantly on compressed inputs which are discarded at inference. Extensive experiments on code language modeling demonstrate that proxy compression substantially improves training efficiency and significantly outperforms pure byte-level baselines given fixed compute budgets. As model scale increases, these gains become more pronounced, and proxy-trained models eventually match or rival tokenizer approaches, all while operating solely on raw bytes and retaining the inherent robustness of byte-level modeling.

翻译：现代语言模型几乎完全在由固定分词器产生的标记序列上进行训练，这种外部无损压缩器通常作用于UTF-8字节序列，从而将模型与该压缩器耦合。本文提出代理压缩，这是一种替代训练方案，在保持压缩输入效率优势的同时，在推理时提供端到端的原始字节接口。训练过程中，一个语言模型在原始字节序列和外部压缩器生成的压缩视图上联合训练；通过此过程，模型学会在内部对齐压缩序列与原始字节。这种对齐实现了两种格式间的强迁移能力，即使主要训练数据为压缩输入（推理时被丢弃）。在代码语言建模上的大量实验表明，代理压缩显著提升了训练效率，并在固定计算预算下明显优于纯字节级基线方法。随着模型规模增大，这些收益变得更加显著，代理训练的模型最终达到或媲美分词器方法，同时仅操作原始字节并保留了字节级建模固有的鲁棒性。