Pretrained language models (PLMs) have shown marvelous improvements across various NLP tasks. Most Chinese PLMs simply treat an input text as a sequence of characters, and completely ignore word information. Although Whole Word Masking can alleviate this, the semantics in words is still not well represented. In this paper, we revisit the segmentation granularity of Chinese PLMs. We propose a mixed-granularity Chinese BERT (MigBERT) by considering both characters and words. To achieve this, we design objective functions for learning both character and word-level representations. We conduct extensive experiments on various Chinese NLP tasks to evaluate existing PLMs as well as the proposed MigBERT. Experimental results show that MigBERT achieves new SOTA performance on all these tasks. Further analysis demonstrates that words are semantically richer than characters. More interestingly, we show that MigBERT also works with Japanese. Our code and model have been released here~\footnote{https://github.com/xnliang98/MigBERT}.

翻译：预训练语言模型（PLMs）已在各类自然语言处理任务中展现出显著性能提升。当前多数中文预训练语言模型简单地将输入文本视为字符序列，完全忽略了词信息。尽管全词掩码策略能部分缓解此问题，但词语蕴含的语义仍未被充分表征。本文重新审视中文预训练语言模型的分词粒度，提出同时考虑字符与词的混合粒度中文BERT模型（MigBERT）。为此，我们设计了兼顾字符级与词级表征的学习目标函数。通过在多种中文NLP任务上的大量实验，我们系统评估了现有预训练语言模型及所提MigBERT模型。实验结果表明，MigBERT在所有任务上均取得最新最优性能。进一步分析揭示，词语的语义丰富度显著高于字符。更有趣的是，我们证明MigBERT同样适用于日语。相关代码与模型已发布至 https://github.com/xnliang98/MigBERT。