Language models (LMs) have introduced a major paradigm shift in Natural Language Processing (NLP) modeling where large pre-trained LMs became integral to most of the NLP tasks. The LMs are intelligent enough to find useful and relevant representations of the language without any supervision. Perhaps, these models are used to fine-tune typical NLP tasks with significantly high accuracy as compared to the traditional approaches. Conversely, the training of these models requires a massively large corpus that is a good representation of the language. English LMs generally perform better than their other language counterparts, due to the availability of massive English corpora. This work elaborates on the design and development of a large Arabic corpus. It consists of over 500 GB of Arabic cleaned text targeted at improving cross-domain knowledge and downstream generalization capability of large-scale language models. Moreover, the corpus is utilized in the training of a large Arabic LM. In order to evaluate the effectiveness of the LM, a number of typical NLP tasks are fine-tuned. The tasks demonstrate a significant boost from 4.5 to 8.5% when compared to tasks fine-tuned on multi-lingual BERT (mBERT). To the best of my knowledge, this is currently the largest clean and diverse Arabic corpus ever collected.

翻译：语言模型（LMs）在自然语言处理建模领域引发重大范式转变，大规模预训练语言模型已成为大多数自然语言处理任务的核心。这些模型能够自主从语言中提取有用且相关的表征，无需人工监督。与传统方法相比，将这些模型用于微调典型自然语言处理任务时，可实现显著更高的准确率。然而，这类模型的训练需要大规模语料库，以充分代表目标语言。英语语言模型通常优于其他语种模型，这得益于大量英语语料库的可获取性。本文详细阐述了一个大型阿拉伯语语料库的设计与开发过程。该语料库包含超过500GB的纯净阿拉伯语文本文本，旨在提升大规模语言模型的跨领域知识能力与下游任务泛化性能。研究进一步将该语料库应用于大型阿拉伯语语言模型的训练。为评估语言模型的有效性，我们微调了多项典型自然语言处理任务。与基于多语言BERT（mBERT）微调的任务相比，这些任务在性能上实现了4.5%至8.5%的显著提升。据我们所知，这是目前所收集的最大规模、最纯净且多样化的阿拉伯语语料库。