Even though large language models (LLMs) have demonstrated remarkable capability in solving various natural language tasks, the capability of an LLM to follow human instructions is still a concern. Recent works have shown great improvements in the instruction-following capability via additional training for instruction-following tasks. However, the mechanisms responsible for effective instruction-following capabilities remain inadequately understood. Here, we introduce a simplified instruction-following task and use synthetic datasets to analyze a Transformer-based causal language model. Our findings suggest that the model learns task-specific information by clustering data within its hidden space, with this clustering process evolving dynamically during learning. We also demonstrate how this phenomenon assists the model in handling unseen instances and validate our results in a more realistic setting.

翻译：尽管大型语言模型（LLMs）在解决各种自然语言任务中展现出显著能力，但其遵循人类指令的能力仍令人担忧。近期研究表明，通过对指令遵循任务进行额外训练，模型在该能力上取得了巨大进步。然而，有效指令遵循能力的潜在机制仍未得到充分理解。本文引入一个简化的指令遵循任务，并使用合成数据集分析基于Transformer的因果语言模型。研究结果表明，模型通过在其隐空间中对数据进行聚类来学习任务特定信息，且该聚类过程在学习过程中动态演化。我们还展示了这一现象如何帮助模型处理未见实例，并在更现实的场景中验证了我们的结论。