Agentic language model (LM) systems power modern applications like "Deep Research" and "Claude Code," and leverage multi-LM architectures to overcome context limitations. Beneath their apparent diversity lies a recurring pattern: smaller "compressor" LMs (that can even run locally) distill raw context into compact text that is then consumed by larger "predictor" LMs. Despite their popularity, the design of compressor-predictor systems remains largely ad hoc, with little guidance on how compressor and predictor choices shape downstream performance. In practice, attributing gains to compression versus prediction requires costly, task-specific pairwise sweeps. We argue that these agentic system design questions are, at root, information-theoretic. Viewing the compressor LM as a noisy channel, we introduce a simple estimator of mutual information between the context and its compression to quantify compression quality in a task-independent way. We show that mutual information strongly predicts downstream performance, independent of any specific task. Through an information-theoretic framework, we perform a comprehensive empirical analysis across five datasets and three model families. Results reveal that larger compressors not only are more accurate, but also more token-efficient, conveying more bits of information per token. A 7B Qwen-2.5 compressor, for instance, is $1.6\times$ more accurate, $4.6\times$ more concise, and conveys $5.5\times$ more bits of mutual information per token than its 1.5B sibling. Across datasets, scaling compressors is substantially more effective than scaling predictors, enabling larger on-device compressors to pair with smaller cloud predictors. Applied to a Deep Research system, these principles enable local compressors as small as 3B parameters to recover $99\%$ of frontier-LM accuracy at $26\%$ of API costs.

翻译：智能体语言模型（LM）系统为"深度研究"和"Claude Code"等现代应用提供支持，其采用多LM架构以突破上下文限制。尽管这些系统表面呈现多样性，其底层存在一种反复出现的模式：较小的"压缩器"LM（甚至可在本地运行）将原始上下文提炼为紧凑文本，随后由较大的"预测器"LM进行解析。尽管此类系统应用广泛，压缩器-预测器系统的设计仍主要依赖经验法则，关于压缩器与预测器选择如何影响下游性能的系统性指导极为缺乏。实践中，区分性能提升源于压缩过程还是预测过程需要进行高成本、任务特定的成对参数扫描。我们认为，这些智能体系统设计问题本质上是信息论问题。通过将压缩器LM视为噪声信道，我们引入一种简单的互信息估计器，用于量化上下文与其压缩表示之间的互信息，从而以任务无关的方式评估压缩质量。我们证明互信息能独立于具体任务地强预测下游性能。基于信息论框架，我们在五个数据集和三个模型族上进行了全面实证分析。结果表明，更大规模的压缩器不仅精度更高，而且具有更高的标记效率，能在每个标记中传递更多比特的信息。例如，与1.5B版本相比，7B参数的Qwen-2.5压缩器精度提升$1.6\times$，压缩率提升$4.6\times$，单位标记传递的互信息比特数增加$5.5\times$。跨数据集分析显示，扩展压缩器规模比扩展预测器规模效果更为显著，这使得更大规模的本地压缩器能够与更小规模的云端预测器协同工作。将上述原理应用于深度研究系统时，仅需3B参数的本地压缩器即可恢复前沿LM模型$99\%$的精度，同时将API成本降低至$26\%$。