The continuous evolution of pre-trained large language models with ever-growing parameters and corpus sizes has augmented their capacity to solve complex tasks. This ability, which obviates the necessity for task-specific training or fine-tuning, relies on providing the model with a language description or some task exemplars -- referred to the prompt -- that guide the desired autoregressive generation. Despite the remarkable success, the underlying mechanisms that facilitate such exceptional generalization abilities remain an open question. In this paper, we present a novel framework that formally conceptualizes answer generation for complex natural language tasks as a hierarchical ``template-content'' structure. According to our modeling, there exist pre-trained models that can automatically decompose tasks into constituent steps during autoregressive generation, through language modeling on a sufficiently large corpus, thereby solving them. Our framework offers an explanatory tool for the complex reasoning abilities of large language models from the perspective of modeling autoregressive generation tasks. Our experiments show that practical models exhibit different behaviors for ``template'' and ``content'' providing support for our modeling.

翻译：持续发展的预训练大型语言模型，随着参数和语料规模的不断增长，增强了其解决复杂任务的能力。这种能力无需针对特定任务进行训练或微调，而是依赖于向模型提供语言描述或任务示例（即提示），从而引导期望的自回归生成。尽管取得了显著成功，但促进这种卓越泛化能力的底层机制仍是一个悬而未决的问题。在本文中，我们提出了一种新颖的框架，将复杂自然语言任务的答案生成正式概念化为一种层次化的“模板-内容”结构。根据我们的建模，存在某些预训练模型能够在自回归生成过程中，通过在足够大的语料上进行语言建模，自动将任务分解为组成步骤，从而解决这些任务。我们的框架从建模自回归生成任务的角度，为大型语言模型的复杂推理能力提供了一种解释工具。实验表明，实际模型在“模板”和“内容”上表现出不同的行为，这为我们的建模提供了支持。