Scaling laws have transformed our understanding of large language models by linking upstream metrics like cross-entropy loss to design factors such as model size, training data, and compute. However, these conventional laws fail to capture downstream task performance, where context plays a critical role. In this work, we propose a straightforward, interpretable framework that jointly models downstream performance as a function of the training compute and the provided context. We empirically validate our framework by fitting it on the observed downstream performance of extended-context variants of Llama-2-7B and Llama-2-13B across 65,500 unique instances spanning three tasks: arithmetic reasoning, common sense reasoning, and machine translation. Our results demonstrate that our framework accurately models in-distribution downstream performance, generalizes across three orders of magnitude in training compute, and reliably extrapolates performance as the amount of context increases. These findings offer valuable insights into the interplay between training compute and context utilization, providing guidance for designing more efficient long-context LLMs for diverse downstream tasks. Our code is available at https://github.com/wang-research-lab/context-scaling.

翻译：缩放定律通过将交叉熵损失等上游指标与模型规模、训练数据和计算量等设计因素相关联，彻底改变了我们对大语言模型的理解。然而，这些传统定律无法捕捉下游任务性能，而上下文在其中起着关键作用。在本研究中，我们提出了一个直观、可解释的框架，将下游性能联合建模为训练计算量和所提供上下文的函数。我们通过对Llama-2-7B和Llama-2-13B扩展上下文变体在三个任务（算术推理、常识推理和机器翻译）上涵盖65,500个独特实例的观测下游性能进行拟合，实证验证了该框架。结果表明，我们的框架能够准确建模分布内下游性能，在三个数量级的训练计算量范围内保持泛化能力，并能可靠地外推上下文量增加时的性能变化。这些发现为理解训练计算量与上下文利用之间的相互作用提供了宝贵见解，为针对多样化下游任务设计更高效的长上下文大语言模型提供了指导。我们的代码发布于 https://github.com/wang-research-lab/context-scaling。