Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT). In this paper, we systematically investigate the advantages and challenges of LLMs for MMT by answering two questions: 1) How well do LLMs perform in translating a massive number of languages? 2) Which factors affect LLMs' performance in translation? We evaluate popular LLMs, including XGLM, OPT, BLOOMZ, and ChatGPT, on 102 languages. Our empirical results show that even the best model ChatGPT still lags behind the supervised baseline NLLB in 83.33% of translation directions. Through further analysis, we discover that LLMs exhibit new working patterns when used for MMT. First, prompt semantics can surprisingly be ignored when given in-context exemplars, where LLMs still show strong performance even with unreasonable prompts. Second, cross-lingual exemplars can provide better task instruction for low-resource translation than exemplars in the same language pairs. Third, we observe the overestimated performance of BLOOMZ on dataset Flores-101, indicating the potential risk when using public datasets for evaluation.

翻译：大型语言模型（LLMs）在处理多语言机器翻译（MMT）方面展现出显著潜力。本文通过回答两个问题系统探究了LLMs在MMT中的优势与挑战：1) LLMs在翻译大量语言时表现如何？2) 哪些因素影响LLMs的翻译性能？我们评估了包括XGLM、OPT、BLOOMZ和ChatGPT在内的主流LLMs在102种语言上的表现。实证结果表明，即使是最优模型ChatGPT，在83.33%的翻译方向上仍落后于有监督基线模型NLLB。进一步分析发现，LLMs在用于MMT时展现出新的工作模式：首先，在提供上下文示例时，提示语义可能被意外忽略，即使使用不合理提示，LLMs仍表现强劲；其次，跨语言示例比同语言对示例能为低资源翻译提供更优的任务指导；第三，我们观察到BLOOMZ在Flores-101数据集上的性能被高估，这表明使用公开数据集进行评估时存在潜在风险。