Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT). In this paper, we systematically investigate the advantages and challenges of LLMs for MMT by answering two questions: 1) How well do LLMs perform in translating massive languages? 2) Which factors affect LLMs' performance in translation? We thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4. Our empirical results show that translation capabilities of LLMs are continually improving. GPT-4 has beat the strong supervised baseline NLLB in 40.91% of translation directions but still faces a large gap towards the commercial translation system, especially on low-resource languages. Through further analysis, we discover that LLMs exhibit new working patterns when used for MMT. First, instruction semantics can surprisingly be ignored when given in-context exemplars. Second, cross-lingual exemplars can provide better task guidance for low-resource translation than exemplars in the same language pairs. Third, LLM can acquire translation ability in a resource-efficient way and generate moderate translation even on zero-resource languages.

翻译：大语言模型（LLMs）在处理多语言机器翻译（MMT）方面展现出显著潜力。本文通过回答两个问题系统探究了LLMs在MMT中的优势与挑战：1）LLMs在翻译海量语言时表现如何？2）哪些因素影响LLMs的翻译性能？我们对包括ChatGPT和GPT-4在内的八种主流LLMs进行了全面评估。实证结果表明，LLMs的翻译能力持续提升。GPT-4在40.91%的翻译方向上已超越强监督基线模型NLLB，但与商业翻译系统仍存在较大差距，尤其在低资源语言上。通过进一步分析，我们发现LLMs在用于MMT时展现出全新的工作模式：其一，当提供上下文示例时，指令语义可能被意外忽略；其二，跨语言示例能为低资源翻译提供比同语言对示例更优的任务引导；其三，LLM能够以资源高效的方式习得翻译能力，甚至在零资源语言上也能生成质量尚可的翻译。