Words change their meaning over time as well as in different contexts. The sense-aware contextualised word embeddings (SCWEs) such as the ones produced by XL-LEXEME by fine-tuning masked langauge models (MLMs) on Word-in-Context (WiC) data attempt to encode such semantic changes of words within the contextualised word embedding (CWE) spaces. Despite the superior performance of SCWEs in contextual/temporal semantic change detection (SCD) benchmarks, it remains unclear as to how the meaning changes are encoded in the embedding space. To study this, we compare pre-trained CWEs and their fine-tuned versions on contextual and temporal semantic change benchmarks under Principal Component Analysis (PCA) and Independent Component Analysis (ICA) transformations. Our experimental results reveal several novel insights such as (a) although there exist a smaller number of axes that are responsible for semantic changes of words in the pre-trained CWE space, this information gets distributed across all dimensions when fine-tuned, and (b) in contrast to prior work studying the geometry of CWEs, we find that PCA to better represent semantic changes than ICA. Source code is available at https://github.com/LivNLP/svp-dims .

翻译：词语的意义会随时间推移及语境变化而发生改变。通过基于上下文词义（WiC）数据对掩码语言模型（MLM）进行微调产生的感知式语境化词嵌入（SCWE，如XL-LEXEME模型所生成的嵌入），试图在语境化词嵌入（CWE）空间中编码此类词语的语义变化。尽管SCWE在语境/时间语义变化检测（SCD）基准测试中表现出优越性能，但意义变化如何在嵌入空间中被编码仍不明确。为探究此问题，我们在主成分分析（PCA）与独立成分分析（ICA）变换框架下，比较了预训练CWE及其在语境与时间语义变化基准上的微调版本。实验结果揭示了若干新发现：（a）尽管预训练CWE空间中存在少量对词语语义变化起主导作用的坐标轴，但微调后该信息会分散至所有维度；（b）与先前研究CWE几何特性的工作相反，我们发现PCA比ICA能更好地表征语义变化。源代码发布于https://github.com/LivNLP/svp-dims。