Training LLMs in low resources languages usually utilizes data augmentation with machine translation (MT) from English language. However, translation brings a number of challenges: there are large costs attached to translating and curating huge amounts of content with high-end machine translation solutions, the translated content carries over cultural biases, and if the translation is not faithful and accurate, the quality of the data degrades causing issues in the trained model. In this work we investigate the role of translation and synthetic data in training language models. We translate TinyStories, a dataset of 2.2M short stories for 3-4 year old children, from English to Arabic using the free NLLB-3B MT model. We train a number of story generation models of sizes 1M-33M parameters using this data. We identify a number of quality and task-specific issues in the resulting models. To rectify these issues, we further pre-train the models with a small dataset of synthesized high-quality stories, representing 1\% of the original training data, using a capable LLM in Arabic. We show using GPT-4 as a judge and dictionary learning analysis from mechanistic interpretability that the suggested approach is a practical means to resolve some of the translation pitfalls. We illustrate the improvement through case studies of linguistic issues and cultural bias.

翻译：在低资源语言中训练大型语言模型通常采用从英语进行机器翻译的数据增强方法。然而，翻译带来了一系列挑战：使用高端机器翻译解决方案翻译和整理海量内容成本高昂，翻译内容会传递文化偏见，且若翻译不忠实、不准确，数据质量下降将导致训练模型出现问题。本研究探讨了翻译与合成数据在语言模型训练中的作用。我们使用免费的NLLB-3B机器翻译模型，将包含220万篇3-4岁儿童短故事的TinyStories数据集从英语翻译为阿拉伯语。基于此数据，我们训练了多个参数量在100万至3300万之间的故事生成模型。我们发现了所得模型中存在的若干质量与任务特定问题。为纠正这些问题，我们使用一个性能优异的阿拉伯语大型语言模型生成的小型高质量合成故事数据集（占原始训练数据的1%），对这些模型进行了进一步预训练。通过采用GPT-4作为评估者以及基于机制可解释性的词典学习分析，我们证明所提出的方法是解决部分翻译缺陷的有效途径。我们通过语言问题和文化偏见的案例研究展示了改进效果。