A better understanding of the emergent computation and problem-solving capabilities of recent large language models is of paramount importance to further improve them and broaden their applicability. This work investigates how a language model, trained to predict the next token, can perform arithmetic computations generalizing beyond training data. Binary addition and multiplication constitute a good testbed for this purpose, since they require a very small vocabulary and exhibit relevant input/output discontinuities making smooth input interpolation ineffective for novel data. We successfully trained a light language model to learn these tasks and ran a number of experiments to investigate the extrapolation capabilities and internal information processing. Our findings support the hypothesis that the language model works as an Encoding-Regression-Decoding machine where the computation takes place in the value space once the input token representation is mapped to an appropriate internal representation.

翻译：理解近期大型语言模型涌现出的计算与问题解决能力，对于进一步优化并拓展其应用领域至关重要。本研究探讨了经过下一标记预测训练的语言模型如何执行泛化能力超越训练数据的算术计算。二进制加法与乘法为此提供了理想的测试平台——这类任务不仅词汇量需求极小，其输入/输出呈现的显著不连续性更使得针对新奇数据的平滑输入插值方法失效。我们成功训练了一个轻量级语言模型掌握这两类任务，并通过系列实验探究其外推能力与内部信息处理机制。研究结果支持以下假设：语言模型本质上是编码-回归-解码机，其计算过程发生在数值空间——当输入标记表征被映射至合适的内部表征后，随即在该空间执行运算。