Language models (LMs) have introduced a major paradigm shift in Natural Language Processing (NLP) modeling where large pre-trained LMs became integral to most of the NLP tasks. The LMs are intelligent enough to find useful and relevant representations of the language without any supervision. Perhaps, these models are used to fine-tune typical NLP tasks with significantly high accuracy as compared to the traditional approaches. Conversely, the training of these models requires a massively large corpus that is a good representation of the language. English LMs generally perform better than their other language counterparts, due to the availability of massive English corpora. This work elaborates on the design and development of a large Arabic corpus. It consists of over 500 GB of Arabic cleaned text targeted at improving cross-domain knowledge and downstream generalization capability of large-scale language models. Moreover, the corpus is utilized in the training of a large Arabic LM. In order to evaluate the effectiveness of the LM, a number of typical NLP tasks are fine-tuned. The tasks demonstrate a significant boost from 4.5 to 8.5% when compared to tasks fine-tuned on multi-lingual BERT (mBERT). To the best of my knowledge, this is currently the largest clean and diverse Arabic corpus ever collected.

翻译：语言模型（LMs）在自然语言处理（NLP）建模领域引发了重大范式转变，使大规模预训练语言模型成为大多数NLP任务的核心组成部分。语言模型具备足够的智能，能在无需监督的情况下发现语言中有用且相关的表征。这些模型可用于微调典型NLP任务，与传统方法相比具有显著更高的准确性。然而，这类模型的训练需要大规模语料库，以便充分代表目标语言。英语语言模型通常优于其他语言的同类模型，这得益于大规模英语语料库的可用性。本研究详细阐述了一个大型阿拉伯语语料库的设计与开发过程。该语料库包含超过500GB的清洗后阿拉伯语文本，旨在提升大规模语言模型的跨领域知识与下游泛化能力。此外，该语料库被用于训练一个大型阿拉伯语语言模型。为评估该语言模型的有效性，我们微调了若干典型NLP任务。与基于多语言BERT（mBERT）微调的同类任务相比，这些任务在性能上实现了4.5%至8.5%的显著提升。据我们所知，这是迄今收集规模最大且质量最高的多元化阿拉伯语语料库。