We propose a new benchmark evaluating the performance of multimodal large language models on rebus puzzles. The dataset covers 333 original examples of image-based wordplay, cluing 13 categories such as movies, composers, major cities, and food. To achieve good performance on the benchmark of identifying the clued word or phrase, models must combine image recognition and string manipulation with hypothesis testing, multi-step reasoning, and an understanding of human cognition, making for a complex, multimodal evaluation of capabilities. We find that proprietary models such as GPT-4V and Gemini Pro significantly outperform all other tested models. However, even the best model has a final accuracy of just 24%, highlighting the need for substantial improvements in reasoning. Further, models rarely understand all parts of a puzzle, and are almost always incapable of retroactively explaining the correct answer. Our benchmark can therefore be used to identify major shortcomings in the knowledge and reasoning of multimodal large language models.

翻译：我们提出一个新的基准，用于评估多模态大型语言模型在谜语（rebus puzzles）上的表现。该数据集包含333个基于图像的文字游戏原创示例，涵盖13个类别，如电影、作曲家、主要城市和食物。要在识别线索词或短语的基准测试中获得良好表现，模型必须将图像识别和字符串操作与假设检验、多步推理以及对人类认知的理解相结合，从而形成对能力的复杂多模态评估。我们发现，GPT-4V和Gemini Pro等专有模型显著优于所有其他测试模型。然而，即使是最佳模型，其最终准确率也仅为24%，凸显了推理能力亟待大幅提升。此外，模型几乎从未能理解谜题的所有部分，且几乎完全无法事后解释正确答案。因此，我们的基准可用于识别多模态大型语言模型在知识和推理方面的主要缺陷。