Multimodal Large Language Models (MLLMs) have demonstrated proficiency in handling a variety of visual-language tasks. However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated. To address this challenge, this paper introduces Mementos, a new benchmark designed to assess MLLMs' sequential image reasoning abilities. Mementos features 4,761 diverse image sequences with varying lengths. We also employ a GPT-4 assisted method to evaluate MLLM reasoning performance. Through a careful evaluation of nine recent MLLMs on Mementos, including GPT-4V and Gemini, we find that they struggle to accurately describe dynamic information about given image sequences, often leading to hallucinations/misrepresentations of objects and their corresponding behaviors. Our quantitative analysis and case studies identify three key factors impacting MLLMs' sequential image reasoning: the correlation between object and behavioral hallucinations, the influence of cooccurring behaviors, and the compounding impact of behavioral hallucinations. Our dataset is available at https://github.com/umd-huang-lab/Mementos.

翻译：多模态大语言模型（MLLMs）已在处理多种视觉-语言任务中展现出显著能力。然而，当前MLLM基准主要设计用于评估基于单张图像静态信息的推理能力，而现代MLLMs从图像序列中推断信息的能力——这对理解我们不断变化的世界至关重要——尚未得到充分研究。为解决这一挑战，本文提出Mementos，这是一个专门评估MLLMs序列图像推理能力的新基准。Mementos包含4,761个长度各异的多样图像序列。我们采用GPT-4辅助方法评估MLLM推理性能。通过对包括GPT-4V和Gemini在内的九个最新MLLMs在Mementos上的仔细评估，发现它们难以准确描述给定图像序列中的动态信息，常导致对物体及其相应行为的幻觉或错误表征。我们的定量分析和案例研究揭示了影响MLLMs序列图像推理的三个关键因素：物体幻觉与行为幻觉之间的相关性、共存行为的影响，以及行为幻觉的复合效应。我们的数据集可在https://github.com/umd-huang-lab/Mementos获取。