This paper aims to benchmark recent progress in language understanding models that output contextualised representations at the character level. Many such modelling architectures and methods to train those architectures have been proposed, but it is currently unclear what the relative contributions of the architecture vs. the pretraining objective are to final model performance. We explore the design space of such models, comparing architectural innovations and a variety of different pretraining objectives on a suite of evaluation tasks with a fixed training procedure in order to find the currently optimal way to build and train character-level BERT-like models. We find that our best performing character-level model exceeds the performance of a token-based model trained with the same settings on the same data, suggesting that character-level models are ready for more widespread adoption. Unfortunately, the best method to train character-level models still relies on a subword-level tokeniser during pretraining, and final model performance is highly dependent on tokeniser quality. We believe our results demonstrate the readiness of character-level models for multilingual language representation, and encourage NLP practitioners to try them as drop-in replacements for token-based models.

翻译：摘要：本文旨在系统评估近期在字符级输出上下文表示的语言理解模型方面的研究进展。许多此类建模架构及相应的训练方法已被提出，但目前尚不清楚架构与预训练目标对最终模型性能的相对贡献。我们探索了此类模型的设计空间，在固定训练流程下，通过一组评估任务比较架构创新与多种不同预训练目标，旨在找出当前构建和训练类BERT字符级模型的最优方案。研究发现，最佳性能的字符级模型超越了基于词元且采用相同设置与数据训练的模型表现，这表明字符级模型已具备更广泛应用的潜力。然而，当前训练字符级模型的最佳方法仍依赖预训练阶段的子词级分词器，且最终模型性能高度依赖于分词器质量。我们相信，本研究结果证明了字符级模型在多语言语言表征中的成熟度，并鼓励自然语言处理从业者将其作为基于词元模型的替代方案进行尝试。