Large Language Models (LLMs) have been adopted and deployed worldwide for a broad variety of applications. However, ensuring their safe use remains a significant challenge. Preference training and safety measures often overfit to harms prevalent in Western-centric datasets, and safety protocols frequently fail to extend to multilingual settings. In this work, we explore model merging in a diverse multi-task setting, combining safety and general-purpose tasks within a multilingual context. Each language introduces unique and varied learning challenges across tasks. We find that objective-based merging is more effective than mixing data, with improvements of up to 8% and 10% in general performance and safety respectively. We also find that language-based merging is highly effective -- by merging monolingually fine-tuned models, we achieve a 4% increase in general performance and 7% reduction in harm across all languages on top of the data mixtures method using the same available data. Overall, our comprehensive study of merging approaches provides a useful framework for building strong and safe multilingual models.

翻译：大型语言模型（LLM）已在全球范围内被广泛采纳并部署于各类应用中。然而，确保其安全使用仍是一项重大挑战。偏好训练与安全措施往往过度拟合以西方为中心的数据集中普遍存在的危害，且安全协议在多语言场景中经常失效。本研究探索了在多样化多任务场景下的模型融合方法，将安全任务与通用任务结合于多语言语境中。每种语言在各项任务中均引入了独特且多样的学习挑战。我们发现，基于目标的模型融合比混合数据更为有效，在通用性能与安全性上分别实现了高达8%和10%的提升。同时，基于语言的融合方法表现出显著效果——通过融合单语言微调模型，我们在使用相同可用数据的数据混合方法基础上，实现了所有语言上通用性能4%的提升与危害率7%的降低。总体而言，我们对融合方法的全面研究为构建强大且安全的多语言模型提供了实用框架。