Language models are defined over a finite set of inputs, which creates a vocabulary bottleneck when we attempt to scale the number of supported languages. Tackling this bottleneck results in a trade-off between what can be represented in the embedding matrix and computational issues in the output layer. This paper introduces PIXEL, the Pixel-based Encoder of Language, which suffers from neither of these issues. PIXEL is a pretrained language model that renders text as images, making it possible to transfer representations across languages based on orthographic similarity or the co-activation of pixels. PIXEL is trained to reconstruct the pixels of masked patches instead of predicting a distribution over tokens. We pretrain the 86M parameter PIXEL model on the same English data as BERT and evaluate on syntactic and semantic tasks in typologically diverse languages, including various non-Latin scripts. We find that PIXEL substantially outperforms BERT on syntactic and semantic processing tasks on scripts that are not found in the pretraining data, but PIXEL is slightly weaker than BERT when working with Latin scripts. Furthermore, we find that PIXEL is more robust than BERT to orthographic attacks and linguistic code-switching, further confirming the benefits of modelling language with pixels.

翻译：语言模型定义在有限输入集上，这限制了可支持语言的规模扩展。克服这一瓶颈需要在嵌入矩阵的表示能力与输出层的计算复杂度之间进行权衡。本文提出像素编码语言模型PIXEL，该模型无需面对上述两难困境。PIXEL是一种预训练语言模型，通过将文本渲染为图像的形式，使得基于字形相似性或像素共激活机制实现跨语言表示迁移成为可能。该模型通过重建被遮蔽像素块的内容进行训练，而非预测词元分布。我们在与BERT相同的英语数据上预训练了8600万参数的PIXEL模型，并在类型学多样的语言（包括多种非拉丁文字体系）的句法与语义任务上进行评估。实验发现，在预训练数据未覆盖的文字体系中，PIXEL在句法与语义处理任务上显著优于BERT；但在处理拉丁文字体系时，PIXEL表现略逊于BERT。此外，PIXEL对字形攻击与语言代码混合的鲁棒性优于BERT，这进一步验证了基于像素建模语言的优势。