Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges. However, the inherent perceptual diversity across languages poses a significant hurdle for multilingual generalization in pixel space. This paper introduces MIXAR, the first generative pixel-based language model trained on eight different languages utilizing a range of different scripts. We empirically evaluate MIXAR against previous pixel-based models as well as comparable tokenizer-based models, demonstrating substantial performance improvement on discriminative and generative multilingual tasks. Additionally, we show how MIXAR is robust to languages never seen during the training. These results are further strengthened when scaling the model to 0.5B parameters which not only improves its capabilities in generative tasks like LAMBADA but also its robustness when challenged with input perturbations such as orthographic attacks.

翻译：基于像素的语言模型正作为传统基于词元方法的替代方案受到关注，其有望规避词元化挑战。然而，语言之间固有的感知差异对像素空间中的多语言泛化构成了重大障碍。本文提出MIXAR，这是首个基于生成式像素的语言模型，该模型使用多种不同文字系统在八种语言上进行训练。我们通过实验将MIXAR与先前基于像素的模型以及同等规模的基于词元器模型进行对比，证明了其在判别式与生成式多语言任务上取得的显著性能提升。此外，我们展示了MIXAR对训练中未见语言的鲁棒性。当将模型扩展至5亿参数时，这些结果得到进一步强化：不仅提升了其在LAMBADA等生成式任务中的能力，还增强了其在面对拼写攻击等输入扰动时的鲁棒性。