The recently released Google Gemini class of models are the first to comprehensively report results that rival the OpenAI GPT series across a wide variety of tasks. In this paper, we do an in-depth exploration of Gemini's language abilities, making two contributions. First, we provide a third-party, objective comparison of the abilities of the OpenAI GPT and Google Gemini models with reproducible code and fully transparent results. Second, we take a closer look at the results, identifying areas where one of the two model classes excels. We perform this analysis over 10 datasets testing a variety of language abilities, including reasoning, answering knowledge-based questions, solving math problems, translating between languages, generating code, and acting as instruction-following agents. From this analysis, we find that Gemini Pro achieves accuracy that is close but slightly inferior to the corresponding GPT 3.5 Turbo on all tasks that we benchmarked. We further provide explanations for some of this under-performance, including failures in mathematical reasoning with many digits, sensitivity to multiple-choice answer ordering, aggressive content filtering, and others. We also identify areas where Gemini demonstrates comparably high performance, including generation into non-English languages, and handling longer and more complex reasoning chains. Code and data for reproduction can be found at https://github.com/neulab/gemini-benchmark

翻译：近期发布的Google Gemini模型系列是首个在广泛任务中全面报告与OpenAI GPT系列相匹敌结果的研究。本文深度探究了Gemini的语言能力，做出两项贡献。首先，我们通过可复现的代码和完全透明的结果，对OpenAI GPT与Google Gemini模型能力进行了第三方客观比较。其次，我们深入分析结果，识别了其中一类模型表现突出的领域。我们在10个数据集上进行分析，测试了包括推理、知识问答、数学解题、语言翻译、代码生成及指令跟随代理在内的多种语言能力。分析发现，Gemini Pro在所有基准测试任务中的准确率虽接近但略低于对应的GPT 3.5 Turbo。我们进一步解释了部分表现不足的原因，包括多位数数学推理失败、对多选题答案顺序敏感、内容过滤过于严格等。同时，我们明确了Gemini表现同等优异的领域，如非英语语言生成、处理更长且更复杂的推理链。复现代码与数据请见https://github.com/neulab/gemini-benchmark