Multi-modal Event Reasoning (MMER) endeavors to endow machines with the ability to comprehend intricate event relations across diverse data modalities. MMER is fundamental and underlies a wide broad of applications. Despite extensive instruction fine-tuning, current multi-modal large language models still fall short in such ability. The disparity stems from that existing models are insufficient to capture underlying principles governing event evolution in various scenarios. In this paper, we introduce Multi-Modal Event Evolution Learning (MEEL) to enable the model to grasp the event evolution mechanism, yielding advanced MMER ability. Specifically, we commence with the design of event diversification to gather seed events from a rich spectrum of scenarios. Subsequently, we employ ChatGPT to generate evolving graphs for these seed events. We propose an instruction encapsulation process that formulates the evolving graphs into instruction-tuning data, aligning the comprehension of event reasoning to humans. Finally, we observe that models trained in this way are still struggling to fully comprehend event evolution. In such a case, we propose the guiding discrimination strategy, in which models are trained to discriminate the improper evolution direction. We collect and curate a benchmark M-EV2 for MMER. Extensive experiments on M-EV2 validate the effectiveness of our approach, showcasing competitive performance in open-source multi-modal LLMs.

翻译：[translated abstract in Chinese] 多模态事件推理（MMER）旨在赋予机器理解跨多样数据模态中复杂事件关系的能力。MMER是基础性任务，并支撑着广泛的应用场景。尽管经过大量的指令微调，当前的多模态大语言模型在此类能力上仍存在不足。这种差距源于现有模型难以捕捉不同场景下支配事件演化的内在规律。本文提出多模态事件演化学习（MEEL），使模型掌握事件演化机制，从而获得先进的MMER能力。具体而言，我们首先设计事件多样化策略，从丰富的场景中采集种子事件；随后采用ChatGPT为这些种子事件生成演化图；提出指令封装流程，将演化图转化为指令微调数据，使模型对事件推理的理解与人类对齐。最后，我们发现以这种方式训练的模型仍难以完全理解事件演化，为此提出引导判别策略，训练模型对非正确演化方向进行判别。我们构建了用于MMER的基准数据集M-EV2，在M-EV2上的大量实验验证了本方法的有效性，并在开源多模态大语言模型中展现出具有竞争力的性能。