Large Language Models (LLMs) are becoming increasingly popular in pervasive computing due to their versatility and strong performance. However, despite their ubiquitous use, the exact mechanisms underlying their outstanding performance remain unclear. Different methods for LLM explainability exist, and many are, as a method, not fully understood themselves. We started with the question of how linguistic abstraction emerges in LLMs, aiming to detect it across different LLM modules (attention heads and input embeddings). For this, we used methods well-established in the literature: (1) probing for token-level relational structures, and (2) feature-mapping using embeddings as carriers of human-interpretable properties. Both attempts failed for different methodological reasons: Attention-based explanations collapsed once we tested the core assumption that later-layer representations still correspond to tokens. Property-inference methods applied to embeddings also failed because their high predictive scores were driven by methodological artifacts and dataset structure rather than meaningful semantic knowledge. These failures matter because both techniques are widely treated as evidence for what LLMs supposedly understand, yet our results show such conclusions are unwarranted. These limitations are particularly relevant in pervasive and distributed computing settings where LLMs are deployed as system components and interpretability methods are relied upon for debugging, compression, and explaining models.

翻译：大型语言模型（LLMs）因其多功能性和强大性能，在普适计算领域日益普及。然而，尽管其应用无处不在，其卓越性能背后的确切机制仍不明确。现有多种LLM可解释性方法，其中许多方法本身作为方法论亦未被完全理解。我们从“语言抽象如何在LLMs中涌现”这一问题出发，试图在不同LLM模块（注意力头与输入嵌入）中检测该现象。为此，我们采用文献中成熟的两种方法：（1）基于词元级关系结构的探测分析；（2）以嵌入向量作为人类可解释属性载体的特征映射。两种尝试均因不同方法论缺陷而失败：基于注意力的解释方法在我们检验“深层表征仍对应词元”这一核心假设时彻底失效；应用于嵌入向量的属性推断方法同样失败，因其高预测分数源于方法论伪影与数据集结构，而非有意义的语义知识。这些失败至关重要，因为这两种技术被广泛视为LLMs理解能力的证据，而我们的结果表明此类结论缺乏依据。这些局限性在普适与分布式计算场景中尤为突出——当LLMs被部署为系统组件时，可解释性方法正被用于模型调试、压缩与解释。