Investigating deep learning language models has always been a significant research area due to the ``black box" nature of most advanced models. With the recent advancements in pre-trained language models based on transformers and their increasing integration into daily life, addressing this issue has become more pressing. In order to achieve an explainable AI model, it is essential to comprehend the procedural steps involved and compare them with human thought processes. Thus, in this paper, we use simple, well-understood non-language tasks to explore these models' inner workings. Specifically, we apply a pre-trained language model to constrained arithmetic problems with hierarchical structure, to analyze their attention weight scores and hidden states. The investigation reveals promising results, with the model addressing hierarchical problems in a moderately structured manner, similar to human problem-solving strategies. Additionally, by inspecting the attention weights layer by layer, we uncover an unconventional finding that layer 10, rather than the model's final layer, is the optimal layer to unfreeze for the least parameter-intensive approach to fine-tune the model. We support these findings with entropy analysis and token embeddings similarity analysis. The attention analysis allows us to hypothesize that the model can generalize to longer sequences in ListOps dataset, a conclusion later confirmed through testing on sequences longer than those in the training set. Lastly, by utilizing a straightforward task in which the model predicts the winner of a Tic Tac Toe game, we identify limitations in attention analysis, particularly its inability to capture 2D patterns.

翻译：探究深度学习语言模型始终是重要研究领域，因其多数先进模型具有“黑箱”特性。随着基于Transformer的预训练语言模型最新进展及其日益融入日常生活，解决这一问题变得更为迫切。为实现可解释的人工智能模型，必须理解模型的处理步骤，并与人类思维过程进行比较。因此，本文采用简单且易于理解的的非语言任务来探索这些模型的内部机制。具体而言，我们将预训练语言模型应用于具有层次结构的约束算术问题，分析其注意力权重分数和隐藏状态。研究表明结果令人鼓舞：模型以类似人类解题策略的方式，以适度结构化的方式处理层次化问题。此外，通过逐层检查注意力权重，我们发现了反直觉的结论：第10层（而非模型最终层）是采用最少参数微调方法的最优解冻层。我们通过熵分析和词嵌入相似性分析支持这些发现。注意力分析使我们提出假设：模型能够泛化至ListOps数据集中更长的序列——该结论随后通过对超出训练集长度的序列进行测试得到证实。最后，通过利用模型预测井字棋游戏获胜者的简单任务，我们识别出注意力分析的局限性，特别是其无法捕获二维模式的问题。