We analyze the capabilities of Transformer language models on learning discrete algorithms. To this end, we introduce two new tasks demanding the composition of several discrete sub-tasks. On both training LLaMA models from scratch and prompting on GPT-4 and Gemini we measure learning compositions of learned primitives. We observe that the compositional capabilities of state-of-the-art Transformer language models are very limited and sample-wise scale worse than relearning all sub-tasks for a new algorithmic composition. We also present a theorem in complexity theory, showing that gradient descent on memorizing feedforward models can be exponentially data inefficient.

翻译：我们分析了Transformer语言模型在学习离散算法方面的能力。为此，我们引入了两个需要组合多个离散子任务的新任务。在从头训练LLaMA模型以及使用GPT-4和Gemini进行提示时，我们测量了学习到的原始操作的组合能力。我们观察到，最先进的Transformer语言模型的组合能力非常有限，并且对于新的算法组合，其样本效率比重新学习所有子任务更差。我们还提出了一个复杂性理论中的定理，表明在记忆前馈模型上的梯度下降可能在数据上呈指数级低效。