We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. Addressing the scarcity of high-quality benchmarks for Basque, we further introduce 4 multiple choice evaluation datasets: EusProficiency, comprising 5,169 questions from official language proficiency exams; EusReading, comprising 352 reading comprehension questions; EusTrivia, comprising 1,715 trivia questions from 5 knowledge areas; and EusExams, comprising 16,774 questions from public examinations. In our extensive evaluation, Latxa outperforms all previous open models we compare to by a large margin. In addition, it is competitive with GPT-4 Turbo in language proficiency and understanding, despite lagging behind in reading comprehension and knowledge-intensive tasks. Both the Latxa family of models, as well as our new pretraining corpora and evaluation datasets, are publicly available under open licenses at https://github.com/hitz-zentroa/latxa. Our suite enables reproducible research on methods to build LLMs for low-resource languages.

翻译：我们提出Latxa——一个参数规模从70亿到700亿的巴斯克语大语言模型系列。Latxa基于Llama 2架构，通过包含430万文档与42亿token的新建巴斯克语语料库进行持续预训练。针对巴斯克语高质量评测基准稀缺的问题，我们进一步构建了4个多项选择评测数据集：EusProficiency（含5,169道官方语言能力测试题）、EusReading（含352道阅读理解题）、EusTrivia（含1,715道涵盖5个知识领域的常识题）和EusExams（含16,774道公开考试题）。在全面的评测中，Latxa以显著优势超越所有此前开源的可比模型。此外，尽管在阅读理解与知识密集型任务上有所不足，该模型在语言能力与理解方面已可与GPT-4 Turbo抗衡。Latxa模型系列、新建预训练语料库及评测数据集均以开源许可协议发布于https://github.com/hitz-zentroa/latxa。我们的套件为构建低资源语言大语言模型提供了可复现的研究方法。