While pretrained language models (PLMs) primarily serve as general-purpose text encoders that can be fine-tuned for a wide variety of downstream tasks, recent work has shown that they can also be rewired to produce high-quality word representations (i.e., static word embeddings) and yield good performance in type-level lexical tasks. While existing work primarily focused on the lexical specialization of monolingual PLMs with immense quantities of monolingual constraints, in this work we expose massively multilingual transformers (MMTs, e.g., mBERT or XLM-R) to multilingual lexical knowledge at scale, leveraging BabelNet as the readily available rich source of multilingual and cross-lingual type-level lexical knowledge. Concretely, we use BabelNet's multilingual synsets to create synonym pairs (or synonym-gloss pairs) across 50 languages and then subject the MMTs (mBERT and XLM-R) to a lexical specialization procedure guided by a contrastive objective. We show that such massively multilingual lexical specialization brings substantial gains in two standard cross-lingual lexical tasks, bilingual lexicon induction and cross-lingual word similarity, as well as in cross-lingual sentence retrieval. Crucially, we observe gains for languages unseen in specialization, indicating that multilingual lexical specialization enables generalization to languages with no lexical constraints. In a series of subsequent controlled experiments, we show that the number of specialization constraints plays a much greater role than the set of languages from which they originate.

翻译：尽管预训练语言模型（PLMs）主要作为通用文本编码器，可通过微调适配各类下游任务，但近期研究表明，这些模型还可被重新配置以生成高质量词汇表示（即静态词嵌入），并在词级词汇任务中取得良好表现。现有研究主要聚焦于利用海量单语言约束对单语言PLMs进行词汇特化，而本工作将大规模多语言变换器（MMTs，如mBERT或XLM-R）暴露于大规模多语言词汇知识中，利用BabelNet这一现成的多语言及跨语言词级词汇知识丰富来源。具体而言，我们使用BabelNet的多语言同义词集创建涵盖50种语言的同义词对（或同义词-释义对），并通过对比学习目标引导MMTs（mBERT和XLM-R）进行词汇特化过程。实验表明，这种大规模多语言词汇特化显著提升了两个标准跨语言词汇任务（双语词典归纳和跨语言词汇相似度）以及跨语言句子检索的性能。尤为关键的是，我们在特化过程中未见语言上也观察到性能提升，这表明多语言词汇特化能够泛化至无词汇约束的语言。后续系列控制实验显示，特化约束的数量比其来源语言集合发挥更重要的作用。