Previous work has shown that the representations output by contextual language models are more anisotropic than static type embeddings, and typically display outlier dimensions. This seems to be true for both monolingual and multilingual models, although much less work has been done on the multilingual context. Why these outliers occur and how they affect the representations is still an active area of research. We investigate outlier dimensions and their relationship to anisotropy in multiple pre-trained multilingual language models. We focus on cross-lingual semantic similarity tasks, as these are natural tasks for evaluating multilingual representations. Specifically, we examine sentence representations. Sentence transformers which are fine-tuned on parallel resources (that are not always available) perform better on this task, and we show that their representations are more isotropic. However, we aim to improve multilingual representations in general. We investigate how much of the performance difference can be made up by only transforming the embedding space without fine-tuning, and visualise the resulting spaces. We test different operations: Removing individual outlier dimensions, cluster-based isotropy enhancement, and ZCA whitening. We publish our code for reproducibility.

翻译：先前研究表明，上下文语言模型输出的表示比静态词嵌入更具各向异性，并且通常存在离群维度。这一现象在单语和多语言模型中似乎都成立，但针对多语言情境的研究相对较少。这些离群值产生的原因及其对表示的影响仍是一个活跃的研究领域。我们调查了多个预训练多语言语言模型中的离群维度及其与各向异性的关系。研究聚焦于跨语言语义相似性任务，因其是评估多语言表示的自然任务。具体而言，我们分析了句子表示。在平行资源（并非始终可用）上微调的句子变换器在此任务上表现更优，且其表示更具各向同性。然而，我们的目标是普遍提升多语言表示的质量。我们探究了在不进行微调的情况下，仅通过变换嵌入空间能弥补多少性能差异，并对变换后的空间进行可视化。我们测试了不同操作：移除单个离群维度、基于聚类的各向同性增强以及ZCA白化处理。为保障可复现性，我们已公开相关代码。