Understanding transformer-based language models is becoming increasingly crucial, particularly as they play pivotal roles in advancing towards artificial general intelligence. However, language model research faces significant challenges, especially for academic research groups with constrained resources. These challenges include complex data structures, unknown target functions, high computational costs and memory requirements, and a lack of interpretability in the inference process, etc. Drawing a parallel to the use of simple models in scientific research, we propose the concept of an anchor function. This is a type of benchmark function designed for studying language models in learning tasks that follow an "anchor-key" pattern. By utilizing the concept of an anchor function, we can construct a series of functions to simulate various language tasks. The anchor function plays a role analogous to that of mice in diabetes research, particularly suitable for academic research. We demonstrate the utility of the anchor function with an example, revealing two basic operations by attention structures in language models: shifting tokens and broadcasting one token from one position to many positions. These operations are also commonly observed in large language models. The anchor function framework, therefore, opens up a series of valuable and accessible research questions for further exploration, especially for theoretical study.

翻译：理解基于Transformer的语言模型正变得越来越重要，尤其是在它们推动通用人工智能发展的过程中发挥着关键作用。然而，语言模型研究面临重大挑战，尤其对于资源有限的学术研究团队而言。这些挑战包括复杂的数据结构、未知的目标函数、高昂的计算成本和内存需求，以及推理过程缺乏可解释性等。类比科学研究中使用简单模型的做法，我们提出了锚函数的概念。这是一种专门设计用于研究语言模型在学习"锚-键"模式任务中的基准函数类别。利用锚函数的概念，我们可以构建一系列函数来模拟各种语言任务。锚函数在语言模型研究中扮演着类似于糖尿病研究中实验小鼠的角色，特别适用于学术研究。我们通过一个示例展示了锚函数的实用性，揭示了注意力结构在语言模型中的两种基本操作：移位标记以及将一个标记从一个位置广播到多个位置。这些操作在大型语言模型中也是普遍存在的。因此，锚函数框架为后续探索（尤其是理论研究）开辟了一系列有价值且易于开展的研究课题。