A better understanding of the emergent computation and problem-solving capabilities of recent large language models is of paramount importance to further improve them and broaden their applicability. This work investigates how a language model, trained to predict the next token, can perform arithmetic computations generalizing beyond training data. Binary addition and multiplication constitute a good testbed for this purpose, since they require a very small vocabulary and exhibit relevant input/output discontinuities making smooth input interpolation ineffective for novel data. We successfully trained a light language model to learn these tasks and ran a number of experiments to investigate the extrapolation capabilities and internal information processing. Our findings support the hypothesis that the language model works as an Encoding-Regression-Decoding machine where the computation takes place in the value space once the input token representation is mapped to an appropriate internal representation.

翻译：更好地理解近期大型语言模型涌现的计算与问题解决能力，对于进一步改进模型并拓展其应用范围至关重要。本研究探讨了基于下一个词元预测训练的语言模型，如何执行超越训练数据泛化范围的算术计算。二进制加法与乘法为这一研究提供了理想的测试平台：这类任务不仅词汇量极小，且输入/输出存在显著的不连续性，使得针对新数据的平滑输入插值方法难以奏效。我们成功训练了一个轻量级语言模型以完成上述任务，并通过系列实验探究其外推能力与内部信息处理机制。研究结果支持如下假设：语言模型本质上是一种"编码-回归-解码"机制，当输入词元表征被映射至恰当的内部表征后，数学计算将在数值空间中完成。