Pretrained contextualized text representation models learn an effective representation of a natural language to make it machine understandable. After the breakthrough of the attention mechanism, a new generation of pretrained models have been proposed achieving good performances since the introduction of the Transformer. Bidirectional Encoder Representations from Transformers (BERT) has become the state-of-the-art model for language understanding. Despite their success, most of the available models have been trained on Indo-European languages however similar research for under-represented languages and dialects remains sparse. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for under represented languages, with a specific focus on the Tunisian dialect. We evaluate our language model on sentiment analysis task, dialect identification task and reading comprehension question-answering task. We show that the use of noisy web crawled data instead of structured data (Wikipedia, articles, etc.) is more convenient for such non-standardized language. Moreover, results indicate that a relatively small web crawled dataset leads to performances that are as good as those obtained using larger datasets. Finally, our best performing TunBERT model reaches or improves the state-of-the-art in all three downstream tasks. We release the TunBERT pretrained model and the datasets used for fine-tuning.

翻译：未经培训的背景文本代表模型学会了一种有效的自然语言代表,使其机器可以理解。在关注机制突破后,提出了新一代经过培训的模型,自采用变异器以来取得了良好的表现。来自变异器(BERT)的双向编码器演示已成为最先进的语言理解模式。尽管这些模型取得了成功,但大多数现有模型都接受了关于印欧语言的培训,然而,对代表性不足的语言和方言的类似研究仍然很少。在本文中,我们调查了为代表性不足的语言培训单一语言变异器语言模型的可行性,具体侧重于突尼斯方言。我们评估了我们关于情绪分析任务、方言识别任务和理解解答任务的语言模型。我们表明,使用噪音的网络爬动数据而不是结构化数据(Wikiepedia、文章等)对于这种非标准化的语言更为方便。此外,结果显示,相对较小的网络爬动数据集导致的性能与使用较大数据集所获得的效果一样好。最后,我们最佳的TUNBERD模型用于所有已使用的下游模型。