We introduce a multilabel probing task to assess the morphosyntactic representations of word embeddings from multilingual language models. We demonstrate this task with multilingual BERT (Devlin et al., 2018), training probes for seven typologically diverse languages of varying morphological complexity: Afrikaans, Croatian, Finnish, Hebrew, Korean, Spanish, and Turkish. Through this simple but robust paradigm, we show that multilingual BERT renders many morphosyntactic features easily and simultaneously extractable (e.g., gender, grammatical case, pronominal type). We further evaluate the probes on six "held-out" languages in a zero-shot transfer setting: Arabic, Chinese, Marathi, Slovenian, Tagalog, and Yoruba. This style of probing has the added benefit of revealing the linguistic properties that language models recognize as being shared across languages. For instance, the probes performed well on recognizing nouns in the held-out languages, suggesting that multilingual BERT has a conception of noun-hood that transcends individual languages; yet, the same was not true of adjectives.

翻译：我们引入了多标签检验任务来评估多语言模式中嵌入的字的形态特征。 我们用多语言BERT(Devlin等人,2018年)来展示这一任务,为七种形式复杂多样的典型语言(南非荷兰语、克罗地亚语、芬兰语、希伯来语、韩语、西班牙语和土耳其语)提供培训探测仪。 通过这一简单而有力的范例,我们显示多语言BERT使许多形态特征易于同时提取(例如性别、语法案例、标语类型) 。 我们进一步评估了六种“持有”语言的探测仪,在零速传输设置中:阿拉伯文、中文、马拉地语、斯洛文尼亚语、塔加罗语和约鲁巴语。 这种演示式的附加好处是揭示语言模式承认语言在各语言之间共享的语言属性。 例如,多语言测试仪在识别被搁置语言中的名词时,很好地进行了确认,表明多语言BERT具有超越了单个语言的名词概念概念;然而,反动性形容词并不真实。