Multimodal information, together with our knowledge, help us to understand the complex and dynamic world. Large language models (LLM) and large multimodal models (LMM), however, still struggle to emulate this capability. In this paper, we present WorldQA, a video understanding dataset designed to push the boundaries of multimodal world models with three appealing properties: (1) Multimodal Inputs: The dataset comprises 1007 question-answer pairs and 303 videos, necessitating the analysis of both auditory and visual data for successful interpretation. (2) World Knowledge: We identify five essential types of world knowledge for question formulation. This approach challenges models to extend their capabilities beyond mere perception. (3) Long-Chain Reasoning: Our dataset introduces an average reasoning step of 4.45, notably surpassing other videoQA datasets. Furthermore, we introduce WorldRetriever, an agent designed to synthesize expert knowledge into a coherent reasoning chain, thereby facilitating accurate responses to WorldQA queries. Extensive evaluations of 13 prominent LLMs and LMMs reveal that WorldRetriever, although being the most effective model, achieved only 70% of humanlevel performance in multiple-choice questions. This finding highlights the necessity for further advancement in the reasoning and comprehension abilities of models. Our experiments also yield several key insights. For instance, while humans tend to perform better with increased frames, current LMMs, including WorldRetriever, show diminished performance under similar conditions. We hope that WorldQA,our methodology, and these insights could contribute to the future development of multimodal world models.

翻译：多模态信息与我们的知识共同帮助我们理解复杂且动态的世界。然而，大型语言模型（LLM）和大型多模态模型（LMM）仍难以模拟这一能力。本文提出视频理解数据集WorldQA，旨在通过三个引人注目的特性推动多模态世界模型的边界：（1）多模态输入：该数据集包含1007个问答对和303个视频，要求同时分析听觉和视觉数据以实现成功解读。（2）世界知识：我们识别了问题构建所需的五种基本世界知识类型，这一方法挑战模型将能力从单纯的感知层面拓展至更深层次。（3）长链推理：本数据集引入平均4.45步的推理步骤，显著超过其他视频问答数据集。此外，我们提出智能体WorldRetriever，旨在将专家知识综合为连贯的推理链，从而促进对WorldQA查询的准确响应。对13个主流LLM和LMM的广泛评估显示，尽管WorldRetriever是最有效的模型，其在多项选择题中仅达到人类水平的70%。这一发现凸显了模型的推理与理解能力仍需进一步推进。我们的实验还获得若干关键见解：例如，人类倾向于在更多帧数下表现更好，而当前LMM（包括WorldRetriever）在类似条件下表现下降。我们期望WorldQA、我们的方法以及这些见解能为未来多模态世界模型的发展做出贡献。