Indonesian language is spoken by almost 200 million people and is the 10th most spoken language in the world, but it is under-represented in NLP (Natural Language Processing) research. A sparsity of language resources has hampered previous work on Indonesian. The Transformer is a new architecture rapidly becoming dominant for NLP, surpassing alternatives like convolutional and recurrent neural networks. T5 (Text-to-Text Transfer Transformer) is a Transformer model that converts all text-based language problems to text-to-text format for English. The multilingual variant is mT5 (multilingual T5) which has shown promising results on many NLP tasks across languages. However, the size of this multilingual model is a drawback for its application in real production applications, which sometimes require only one language. In this study, the mT5 model was adapted for only one language, Indonesian, resulting in a pre-trained T5 model that was specific only for Indonesian with a smaller size. For performance comparison, we fine-tuned this model and the mT5 model to the Sentiment Analysis (SA), Question Generation (QG), and Question Answering (QA) tasks with the exact mechanism and dataset. Fine-tuned model based on our model achieved 77.18% accuracy on SA, 8% higher than the mT5-based model, and obtained nearly the same score as the mT5-based model on QG and QA. The results confirm that it is possible to produce a smaller pre-trained model that maintains comparable yields while reducing the model size by up to 58%. In addition, the resulting model requires less memory, loads faster, and inference times faster.

翻译：印尼语拥有近两亿使用者，是世界第十大语言，但在自然语言处理（NLP）研究中代表性不足。语言资源的匮乏阻碍了此前针对印尼语的研究工作。Transformer是一种迅速成为NLP主流的新架构，性能超越卷积神经网络和循环神经网络等替代方案。T5（文本到文本迁移Transformer）是一种将文本类语言问题转化为文本到文本格式的Transformer模型，目前主要针对英语。其多语言变体mT5（多语言T5）在跨语言NLP任务中展现出良好效果。然而，此类多语言模型规模较大，在实际生产应用中（有时仅需单语言支持）存在局限。本研究将mT5模型适配为仅支持印尼语的单语言模型，生成了更小巧的印尼语专用预训练T5模型。为进行性能对比，我们采用相同机制和数据集，将该模型与mT5模型分别微调至情感分析（SA）、问题生成（QG）和问答（QA）任务。基于本模型的微调版本在SA任务上达到77.18%的准确率，较mT5模型提升8%；在QG和QA任务上得分与mT5模型近乎持平。结果表明，在将模型规模缩减高达58%的前提下，仍可生成保持可比性能的更小预训练模型。此外，所生成的模型内存占用更少、加载速度更快、推理时间更短。