Generative AI models have impressive performance on many Natural Language Processing tasks such as language understanding, reasoning and language generation. One of the most important questions that is being asked by the AI community today is about the capabilities and limits of these models, and it is clear that evaluating generative AI is very challenging. Most studies on generative Large Language Models (LLMs) are restricted to English and it is unclear how capable these models are at understanding and generating other languages. We present the first comprehensive benchmarking of generative LLMs - MEGA, which evaluates models on standard NLP benchmarks, covering 8 diverse tasks and 33 typologically diverse languages. We also compare the performance of generative LLMs to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs. We present a thorough analysis of the performance of models across languages and discuss some of the reasons why generative LLMs are currently not optimal for all languages. We create a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.

翻译：摘要：生成式人工智能模型在语言理解、推理和语言生成等众多自然语言处理任务中展现出卓越性能。当前人工智能界关注的核心问题之一是如何界定这类模型的能力边界，而评估生成式人工智能显然极具挑战性。现有关于生成式大语言模型（LLMs）的研究多局限于英语，尚不清楚这些模型在其他语言理解与生成任务中的表现。我们首次提出了生成式大语言模型的综合基准测试——MEGA，该测试覆盖了8个不同任务和33种类型多样的语言，在标准NLP基准上评估模型性能。同时，我们通过对比生成式大语言模型与当前最先进的（SOTA）非自回归模型在这些任务上的表现，揭示了生成模型相对于前代大语言模型的性能差异。我们系统分析了模型在不同语言上的性能表现，并探讨了当前生成式大语言模型尚无法适用于所有语言的部分原因。本研究建立了多语言环境下评估生成式大语言模型的框架，为领域未来发展指明了方向。