We analyze the capabilities of Transformer language models on learning discrete algorithms. To this end, we introduce two new tasks demanding the composition of several discrete sub-tasks. On both training LLaMA models from scratch and prompting on GPT-4 and Gemini we measure learning compositions of learned primitives. We observe that the compositional capabilities of state-of-the-art Transformer language models are very limited and sample-wise scale worse than relearning all sub-tasks for a new algorithmic composition. We also present a theorem in complexity theory, showing that gradient descent on memorizing feedforward models can be exponentially data inefficient.

翻译：我们分析了Transformer语言模型在学习离散算法方面的能力。为此，我们引入了两个需要组合多个离散子任务的新任务。通过对从头训练的LLaMA模型以及基于GPT-4和Gemini的提示学习进行实验，我们衡量了模型对已学习原语的组合能力。我们观察到，当前最先进的Transformer语言模型的组合能力非常有限，且随着样本量的增加，其表现甚至劣于针对新算法组合重新学习所有子任务。我们还给出了复杂性理论中的一个定理，表明在记忆型前馈模型上的梯度下降算法可能在数据效率上呈指数级低下。