A better understanding of the emergent computation and problem-solving capabilities of recent large language models is of paramount importance to further improve them and broaden their applicability. This work investigates how a language model, trained to predict the next token, can perform arithmetic computations generalizing beyond training data. Binary addition and multiplication constitute a good testbed for this purpose, since they require a very small vocabulary and exhibit relevant input/output discontinuities making smooth input interpolation ineffective for novel data. We successfully trained a light language model to learn these tasks and ran a number of experiments to investigate the extrapolation capabilities and internal information processing. Our findings support the hypotheses that the language model works as an Encoding-Regression-Decoding machine where the computation takes place in the value space once the input token representation is mapped to an appropriate internal representation.

翻译：近期大型语言模型涌现出的计算与问题解决能力，对于进一步优化模型并拓展其应用场景具有至关重要的研究价值。本研究探讨了经过下一个词元预测训练的语言模型，如何执行超越训练数据的算术运算泛化任务。二进制加法与乘法因其词汇表规模极小且存在显著的输入/输出不连续性（使得平滑输入插值对新型数据无效），成为验证该能力的理想实验平台。我们成功训练了一个轻量级语言模型掌握上述任务，并开展系列实验探究其外推能力与内部信息处理机制。研究结果支持以下假设：语言模型本质上是编码-回归-解码器，当输入词元表征被映射至合适的内部表征后，计算过程在数值空间中完成。