This paper presents a study on strategies to enhance the translation capabilities of large language models (LLMs) in the context of machine translation (MT) tasks. The paper proposes a novel paradigm consisting of three stages: Secondary Pre-training using Extensive Monolingual Data, Continual Pre-training with Interlinear Text Format Documents, and Leveraging Source-Language Consistent Instruction for Supervised Fine-Tuning. Previous research on LLMs focused on various strategies for supervised fine-tuning (SFT), but their effectiveness has been limited. While traditional machine translation approaches rely on vast amounts of parallel bilingual data, our paradigm highlights the importance of using smaller sets of high-quality bilingual data. We argue that the focus should be on augmenting LLMs' cross-lingual alignment abilities during pre-training rather than solely relying on extensive bilingual data during SFT. Experimental results conducted using the Llama2 model, particularly on Chinese-Llama2 after monolingual augmentation, demonstrate the improved translation capabilities of LLMs. A significant contribution of our approach lies in Stage2: Continual Pre-training with Interlinear Text Format Documents, which requires less than 1B training data, making our method highly efficient. Additionally, in Stage3, we observed that setting instructions consistent with the source language benefits the supervised fine-tuning process. Experimental results demonstrate that our approach surpasses previous work and achieves superior performance compared to models such as NLLB-54B and GPT3.5-text-davinci-003, despite having a significantly smaller parameter count of only 7B or 13B. This achievement establishes our method as a pioneering strategy in the field of machine translation.

翻译：本文研究了在机器翻译（MT）任务中提升大语言模型（LLMs）翻译能力的策略。我们提出了一种包含三个阶段的新范式：使用大规模单语数据进行二次预训练、使用对照文本格式文档进行持续预训练，以及利用源语言一致指令进行监督微调。以往针对LLM的研究聚焦于各种监督微调（SFT）策略，但其效果有限。传统机器翻译方法依赖海量的平行双语数据，而我们的范式则强调使用少量高质量双语数据的重要性。我们认为，重点应放在预训练阶段增强LLM的跨语言对齐能力，而非仅依赖SFT阶段的大量双语数据。基于Llama2模型（特别是经单语增强后的中文-Llama2）的实验结果表明，该范式显著提升了LLM的翻译能力。我们方法的重要贡献在于第二阶段：使用对照文本格式文档进行持续预训练，该阶段仅需不到1B的训练数据，具有极高的效率。此外，在第三阶段，我们观察到设置与源语言一致的指令有利于监督微调过程。实验结果显示，尽管我们的方法仅使用7B或13B参数（远小于其他模型），但其性能超越了NLLB-54B和GPT3.5-text-davinci-003等模型，且优于现有工作。这一成果确立了我们的方法在机器翻译领域中的开创性地位。