Multilingual machine translation has proven immensely useful for both parameter efficiency and overall performance across many language pairs via complete multilingual parameter sharing. However, some language pairs in multilingual models can see worse performance than in bilingual models, especially in the one-to-many translation setting. Motivated by their empirical differences, we examine the geometric differences in representations from bilingual models versus those from one-to-many multilingual models. Specifically, we compute the isotropy of these representations using intrinsic dimensionality and IsoScore, in order to measure how the representations utilize the dimensions in their underlying vector space. Using the same evaluation data in both models, we find that for a given language pair, its multilingual model decoder representations are consistently less isotropic and occupy fewer dimensions than comparable bilingual model decoder representations. Additionally, we show that much of the anisotropy in multilingual decoder representations can be attributed to modeling language-specific information, therefore limiting remaining representational capacity.

翻译：多语言机器翻译通过完全的多语言参数共享，在参数效率和跨语言对的整体性能方面已被证明极为有效。然而，多语言模型中的某些语言对的性能可能低于双语模型，尤其是在一对多翻译场景中。受其经验差异的启发，我们研究了双语模型与一对多多语言模型在表示上的几何差异。具体而言，我们使用内在维度和IsoScore计算这些表示的等向性，以衡量表示如何利用其底层向量空间中的维度。使用相同的评估数据对两种模型进行测试，我们发现对于给定的语言对，其多语言模型解码器表示的等向性始终低于同等双语模型解码器表示，并且占据的维度更少。此外，我们表明多语言解码器表示中的许多各向异性可归因于对语言特定信息的建模，从而限制了剩余的表示容量。