Aligning language models (LMs) based on human-annotated preference data is a crucial step in obtaining practical and performant LM-based systems. However, multilingual human preference data are difficult to obtain at scale, making it challenging to extend this framework to diverse languages. In this work, we evaluate a simple approach for zero-shot cross-lingual alignment, where a reward model is trained on preference data in one source language and directly applied to other target languages. On summarization and open-ended dialog generation, we show that this method is consistently successful under comprehensive evaluation settings, including human evaluation: cross-lingually aligned models are preferred by humans over unaligned models on up to >70% of evaluation instances. We moreover find that a different-language reward model sometimes yields better aligned models than a same-language reward model. We also identify best practices when there is no language-specific data for even supervised finetuning, another component in alignment.

翻译：基于人工标注偏好数据对齐语言模型是获得实用且高效的语言模型系统的关键步骤。然而，多语言人工偏好数据难以大规模获取，这使得将该框架扩展到多种语言变得具有挑战性。在本工作中，我们评估了一种用于零样本跨语言对齐的简单方法：在一种源语言的偏好数据上训练奖励模型，并直接应用于其他目标语言。在摘要生成和开放式对话生成任务中，我们展示了该方法在包括人工评估在内的全面评估设置下始终成功：跨语言对齐模型在多达70%以上的评估实例中被人类偏好于未对齐模型。此外，我们发现，不同语言的奖励模型有时比同语言的奖励模型能产生更优的对齐模型。我们还确定了在缺乏特定语言数据时（即使是监督微调，作为对齐的另一组成部分）的最佳实践。