Training large language models (LLMs) in low-resource languages such as Hebrew poses unique challenges. In this paper, we introduce DictaLM2.0 and DictaLM2.0-Instruct, two LLMs derived from the Mistral model, trained on a substantial corpus of approximately 200 billion tokens in both Hebrew and English. Adapting a pre-trained model to a new language involves specialized techniques that differ significantly from training a model from scratch or further training existing models on well-resourced languages such as English. We outline these novel training methodologies, which facilitate effective learning and adaptation to the linguistic properties of Hebrew. Additionally, we fine-tuned DictaLM2.0-Instruct on a comprehensive instruct dataset to enhance its performance on task-specific instructions. To rigorously evaluate our models, we introduce a new benchmark suite for Hebrew LLM evaluation, covering a diverse set of tasks including Question Answering, Sentiment Analysis, Winograd Schema Challenge, Translation, and Summarization. Our work not only addresses the intricacies of training LLMs in low-resource languages but also proposes a framework that can be leveraged for adapting other LLMs to various non-English languages, contributing to the broader field of multilingual NLP.

翻译：在低资源语言（如希伯来语）中训练大型语言模型（LLMs）面临独特挑战。本文介绍了DictaLM2.0与DictaLM2.0-Instruct，这两个基于Mistral模型衍生的LLMs，在约2000亿词元的希伯来语与英语混合语料库上进行了大规模训练。将预训练模型适配至新语言需要采用与从头训练模型或在英语等高资源语言上进一步训练现有模型显著不同的专门技术。我们概述了这些新颖的训练方法，它们促进了模型对希伯来语语言特性的有效学习与适应。此外，我们在全面的指令数据集上对DictaLM2.0-Instruct进行了微调，以提升其在任务特定指令上的性能。为严格评估我们的模型，我们引入了一套新的希伯来语LLM评估基准测试集，涵盖问答、情感分析、Winograd模式挑战、翻译与摘要等多种任务。我们的工作不仅解决了在低资源语言中训练LLMs的复杂性，还提出了一个可推广的框架，用于将其他LLMs适配至各种非英语语言，为多语言NLP的更广泛领域做出贡献。