Pretrained character-level and byte-level language models have been shown to be competitive with popular subword models across a range of Natural Language Processing (NLP) tasks. However, there has been little research on their effectiveness for neural machine translation (NMT), particularly within the popular pretrain-then-finetune paradigm. This work performs an extensive comparison across multiple languages and experimental conditions of character- and subword-level pretrained models (ByT5 and mT5, respectively) on NMT. We show the effectiveness of character-level modeling in translation, particularly in cases where fine-tuning data is limited. In our analysis, we show how character models' gains in translation quality are reflected in better translations of orthographically similar words and rare words. While evaluating the importance of source texts in driving model predictions, we highlight word-level patterns within ByT5, suggesting an ability to modulate word-level and character-level information during generation. We conclude by assessing the efficiency tradeoff of byte models, suggesting their usage in non-time-critical scenarios to boost translation quality.

翻译：预训练的字符级和字节级语言模型已被证明在多种自然语言处理（NLP）任务中与流行的子词模型具有竞争力。然而，关于它们在神经机器翻译（NMT）中的有效性，尤其是在流行的预训练-微调范式下，研究尚不充分。本文在多种语言和实验条件下，对字符级和子词级预训练模型（分别为ByT5和mT5）在NMT中的表现进行了广泛比较。我们展示了字符级建模在翻译中的有效性，特别是在微调数据有限的情况下。分析中，我们揭示了字符模型在翻译质量上的提升如何体现为对拼写相似词和罕见词的更好翻译。在评估源文本对驱动模型预测的重要性时，我们突出了ByT5中的词级模式，表明其在生成过程中能够调节词级和字符级信息。最后，我们评估了字节模型的效率权衡，建议在非时间关键场景中使用它们以提升翻译质量。