In this study, we propose an axiomatic system to define and quantify the precise memorization and in-context reasoning effects used by the large language model (LLM) for language generation. These effects are formulated as non-linear interactions between tokens/words encoded by the LLM. Specifically, the axiomatic system enables us to categorize the memorization effects into foundational memorization effects and chaotic memorization effects, and further classify in-context reasoning effects into enhanced inference patterns, eliminated inference patterns, and reversed inference patterns. Besides, the decomposed effects satisfy the sparsity property and the universal matching property, which mathematically guarantee that the LLM's confidence score can be faithfully decomposed into the memorization effects and in-context reasoning effects. Experiments show that the clear disentanglement of memorization effects and in-context reasoning effects enables a straightforward examination of detailed inference patterns encoded by LLMs.

翻译：本研究提出一个公理系统，用于定义并量化大语言模型在语言生成过程中所利用的精确记忆效应与上下文推理效应。这些效应被形式化为大语言模型编码的词元/词语间的非线性交互作用。具体而言，该公理系统使我们能够将记忆效应划分为基础记忆效应与混沌记忆效应，并进一步将上下文推理效应分类为增强推理模式、消除推理模式与反转推理模式。此外，分解后的效应满足稀疏性与通用匹配性，从数学上保证大语言模型的置信度得分可忠实分解为记忆效应与上下文推理效应。实验表明，记忆效应与上下文推理效应清晰解耦后，可直接检验大语言模型编码的详细推理模式。