We propose ContextLM, a framework that implicitly learns multi-token prediction by augmenting standard pretraining with an intrinsic next-context prediction objective. ContextLM builds a language model on top of context embeddings that span multiple tokens, enabling better next-token prediction by predicting the next context. Our model is fully compatible with standard autoregressive, token-by-token evaluation paradigms (e.g., perplexity). Extensive experiments with GPT-2 and Pythia backbones (up to 1.5B parameters and 300B training tokens) reveal that ContextLM shifts the Pareto frontier of scaling laws, exhibiting superior efficiency in parameters, training tokens, and FLOPs. Our results show that ContextLM could already achieve the baseline perplexity using 39\% fewer parameters and demonstrates robust generalization improvements on extensive downstream tasks under equivalent parameter counts.

翻译：我们提出ContextLM框架，该框架通过为标准预训练增加内在的下一上下文预测目标，隐式学习多标记预测。ContextLM在跨越多个标记的上下文嵌入基础上构建语言模型，通过预测下一上下文实现更优的下一标记预测。本模型完全兼容标准的自回归逐标记评估范式（如困惑度）。基于GPT-2和Pythia主干网络（最高达15亿参数和3000亿训练标记）的大规模实验表明，ContextLM改变了缩放定律的帕累托前沿，在参数量、训练标记数和浮点运算次数方面均展现出更优效率。实验结果显示，ContextLM仅需减少39%的参数即可达到基线困惑度，并在同等参数量条件下在广泛下游任务中表现出稳健的泛化性能提升。