Subword tokenization schemes are the dominant technique used in current NLP models. However, such schemes can be rigid and tokenizers built on one corpus do not adapt well to other parallel corpora. It has also been observed that in multilingual corpora, subword tokenization schemes over-segment low-resource languages leading to a drop in translation performance. A simple alternative to subword tokenizers is byte-based methods i.e. tokenization into byte sequences using encoding schemes such as UTF-8. Byte tokens often represent inputs at a sub-character granularity i.e. one character can be represented by a sequence of multiple byte tokens. This results in byte sequences that are significantly longer than character sequences. Enforcing aggregation of local information in the lower layers can guide the model to build higher-level semantic information. We propose a Local Byte Fusion (LOBEF) method for byte-based machine translation -- utilizing byte $n$-gram and word boundaries -- to aggregate local semantic information. Extensive experiments on multilingual translation, zero-shot cross-lingual transfer, and domain adaptation reveal a consistent improvement over traditional byte-based models and even over subword techniques. Further analysis also indicates that our byte-based models are parameter-efficient and can be trained faster than subword models.

翻译：子词分词方案是当前自然语言处理模型中的主流技术。然而，此类方案往往较为僵化，且基于某个语料库构建的分词器难以良好适配其他平行语料库。现有研究还发现，在多语言语料库中，子词分词方案对低资源语言会产生过度切分现象，导致翻译性能下降。字节级方法（例如利用UTF-8等编码方案将文本切分为字节序列）是子词分词器的简单替代方案。字节级词元通常以亚字符粒度（即一个字符可能由多个字节词元组成的序列表示）表征输入，这导致字节序列长度显著长于字符序列。在低层网络中强制进行局部信息聚合，可引导模型构建更高层次的语义信息。为此，我们提出面向字节级机器翻译的局部字节融合方法（LOcal Byte Fusion, LOBEF）——利用字节n-gram与词边界信息——聚合局部语义信息。针对多语言翻译、零样本跨语言迁移及领域适配的大量实验表明，该方法相较于传统字节级模型乃至子词技术均取得了一致性提升。进一步分析还显示，我们的字节级模型具有参数高效性，且训练速度优于子词模型。