Do neural networks, trained on well-understood algorithmic tasks, reliably rediscover known algorithms for solving those tasks? Several recent studies, on tasks ranging from group arithmetic to in-context linear regression, have suggested that the answer is yes. Using modular addition as a prototypical problem, we show that algorithm discovery in neural networks is sometimes more complex. Small changes to model hyperparameters and initializations can induce the discovery of qualitatively different algorithms from a fixed training set, and even parallel implementations of multiple such algorithms. Some networks trained to perform modular addition implement a familiar Clock algorithm; others implement a previously undescribed, less intuitive, but comprehensible procedure which we term the Pizza algorithm, or a variety of even more complex procedures. Our results show that even simple learning problems can admit a surprising diversity of solutions, motivating the development of new tools for characterizing the behavior of neural networks across their algorithmic phase space.

翻译：神经网络在训练于已知算法任务时，是否总能可靠地发现解决这些任务的已知算法？近期针对从群算术到上下文线性回归等任务的多项研究表明，答案是肯定的。我们以模加法为典型问题，证明神经网络中的算法发现有时更为复杂。模型超参数和初始化的微小变化，足以使同一固定训练集诱导出性质完全不同的算法发现，甚至并行实现多种此类算法。部分训练执行模加法的网络实现了熟悉的时钟算法，另一些则实现了此前未见描述、虽不直观但可理解的程序（我们称之为披萨算法），乃至更加复杂的多种程序。我们的结果表明，即便是简单的学习问题也能容纳令人惊讶的算法多样性，这激励着开发新工具来表征神经网络在其算法相空间中的行为。