Modern language models are trained almost exclusively on token sequences produced by a fixed tokenizer, an external lossless compressor often over UTF-8 byte sequences, thereby coupling the model to that compressor. This work introduces proxy compression, an alternative training scheme that preserves the efficiency benefits of compressed inputs while providing an end-to-end, raw-byte interface at inference time. During training, a single language model is jointly trained on raw byte sequences and compressed views generated by external compressors; through the process, the model learns to internally align compressed sequences and raw bytes. This alignment enables strong transfer between the two formats, even when training predominantly on compressed inputs that are discarded at inference. Extensive experiments on code language modeling demonstrate that proxy compression substantially improves training efficiency and significantly outperforms pure byte-level baselines given fixed compute budgets. As model scale increases, these gains become more pronounced, and proxy-trained models eventually match or surpass tokenizer approaches, all while operating solely on raw bytes and retaining the inherent robustness of byte-level modeling. Our code is available at https://github.com/LZhengisme/proxy-compression.

翻译：现代语言模型几乎完全在由固定分词器生成的令牌序列上训练，该分词器通常是对UTF-8字节序列进行无损压缩的外部压缩器，从而将模型与特定压缩器绑定。本文提出了一种替代训练方案——代理压缩，它在保留压缩输入效率优势的同时，在推理阶段提供端到端的原始字节接口。训练过程中，单个语言模型同时在原始字节序列和由外部压缩器生成的压缩视图上联合训练；通过这一过程，模型学会在内部对齐压缩序列与原始字节。这种对齐机制能够实现两种格式之间的强迁移，即使在训练主要依赖推理阶段被丢弃的压缩输入时也是如此。在代码语言建模上的大量实验表明，在固定计算预算下，代理压缩显著提升了训练效率，并大幅超过了纯字节级基线方法。随着模型规模增大，这些增益愈发显著，采用代理压缩训练的模型最终能够匹配甚至超越基于分词器的方法，同时完全基于原始字节运行，并保留了字节级建模固有的鲁棒性。我们的代码发布于 https://github.com/LZhengisme/proxy-compression。