We study the compression of LLM-generated text across lossless and lossy regimes, characterizing a compression-compute frontier where more compression is possible at the cost of more compute. For lossless compression, domain-adapted LoRA adapters can improve LLM-based arithmetic coding by 2x over compression with the base LLM alone. For lossy compression, prompting a model for a succinct rewrite then applying arithmetic coding can achieve compression ratios of approximately 0.03, a 2x improvement over compressing the original response. We further introduce Question-Asking compression (QA), an interactive lossy protocol inspired by the game 'Twenty Questions'. A small model iteratively refines its response by asking yes/no questions to a stronger model, transferring exactly one bit per answer. On 8 benchmarks spanning math, science, and code, 10 binary questions recover 23% to 72% of the capability gap between a small and large model on standard benchmarks and 7% to 38% on harder benchmarks, achieving compression ratios of 0.0006 to 0.004. This is over 100x smaller than prior LLM-based compression (Deletang et al., 2024), suggesting that interactive protocols can transfer knowledge far more efficiently than transmitting full responses.

翻译：我们研究了在无损和有损两种框架下对大语言模型生成文本的压缩，表征了压缩-计算前沿——即通过增加计算量可实现更高压缩率。在无损压缩方面，领域适配的LoRA适配器可使基于LLM的算术编码压缩效率相比仅使用基础LLM时提升2倍。在有损压缩方面，通过提示模型生成简洁改写版本再应用算术编码，可实现约0.03的压缩比，相比直接压缩原始响应提升2倍。我们进一步提出了提问式压缩（QA），这是一种受"二十问"游戏启发的交互式有损协议：小模型通过向强模型提出是非问题逐步优化响应，每个答案恰好传输1比特信息。在涵盖数学、科学和代码的8个基准测试中，10个二元问题可恢复小模型与大模型在标准基准上能力差距的23%至72%，在更困难的基准上恢复7%至38%，同时实现0.0006至0.004的压缩比。这比先前基于LLM的压缩方法（Deletang等，2024）小100倍以上，表明交互式协议传递知识的效率远高于直接传输完整响应。