Given the success of Large Language Models (LLMs), there has been considerable interest in studying the properties of model activations. The literature overwhelmingly agrees that LLM representations are dominated by a few ``outlier dimensions'' with exceedingly high variance and magnitude. Several studies in Natural Language Processing (NLP) have sought to mitigate the impact of such outlier dimensions and force LLMs to be isotropic (i.e., have uniform variance across all dimensions in embedding space). Isotropy is thought to be a desirable property for LLMs that improves model performance and more closely aligns textual representations with human intuition. However, many of the claims regarding isotropy in NLP have been based on the average cosine similarity of embeddings, which has recently been shown to be a flawed measure of isotropy. In this paper, we propose I-STAR: IsoScore*-based STable Anisotropic Regularization, a novel regularization method that can be used to increase or decrease levels of isotropy in embedding space during training. I-STAR uses IsoScore*, the first accurate measure of isotropy that is both differentiable and stable on mini-batch computations. In contrast to several previous works, we find that decreasing isotropy in contextualized embeddings improves performance on the majority of tasks and models considered in this paper.

翻译：鉴于大型语言模型（LLMs）的成功，研究模型激活属性引起了广泛关注。文献普遍认为，LLM表示由少数具有极高方差和量级的“异常维度”主导。自然语言处理领域的多项研究试图减轻此类异常维度的影响，促使LLM实现各向同性（即在嵌入空间的所有维度上具有均匀方差）。各向同性被视为LLM的理想特性，可提升模型性能，并使文本表示更符合人类直觉。然而，许多关于NLP中各向同性的主张基于嵌入的平均余弦相似度，而近期研究表明该度量存在缺陷。本文提出I-STAR：基于IsoScore*的稳定各向异性正则化方法，这是一种可在训练过程中灵活增减嵌入空间各向同性水平的新型正则化方法。I-STAR采用IsoScore*——首个兼具可微性与小批量计算稳定性的各向同性精确度量。与以往研究不同，我们发现降低上下文嵌入的各向同性水平在本文所考察的大多数任务和模型上均能提升性能。