Recent advances in video large language models have demonstrated strong capabilities in understanding short clips. However, scaling them to hours- or days-long videos remains highly challenging due to limited context capacity and the loss of critical visual details during abstraction. Existing memory-augmented methods mitigate this by leveraging textual summaries of video segments, yet they heavily rely on text and fail to utilize visual evidence when reasoning over complex scenes. Moreover, retrieving from fixed temporal scales further limits their flexibility in capturing events that span variable durations. To address this, we introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories, encompassing both textual and visual representations. WorldMM comprises three types of memory: episodic memory indexes factual events across multiple temporal scales, semantic memory continuously updates high-level conceptual knowledge, and visual memory preserves detailed information about scenes. During inference, an adaptive retrieval agent iteratively selects the most relevant memory source and leverages multiple temporal granularities based on the query, continuing until it determines that sufficient information has been gathered. WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks, achieving an average 8.4% performance gain over previous state-of-the-art methods, showing its effectiveness on long video reasoning.

翻译：近期视频大语言模型的进展已展现出在理解短视频片段方面的强大能力。然而，由于上下文容量有限以及在抽象过程中关键视觉细节的丢失，将其扩展到数小时甚至数天时长的视频仍极具挑战性。现有的记忆增强方法通过利用视频片段的文本摘要来缓解这一问题，但它们严重依赖文本，在推理复杂场景时未能有效利用视觉证据。此外，从固定的时间尺度进行检索进一步限制了其捕捉不同持续时间事件的灵活性。为解决这些问题，我们提出了WorldMM，一种新颖的多模态记忆智能体，它构建并检索来自多种互补记忆，涵盖文本和视觉表征。WorldMM包含三种类型的记忆：情景记忆索引跨多个时间尺度的事实性事件，语义记忆持续更新高层概念知识，视觉记忆则保留场景的详细信息。在推理过程中，一个自适应检索智能体根据查询，迭代地选择最相关的记忆源并利用多时间粒度，直至确定已收集足够信息为止。WorldMM在五个长视频问答基准测试中显著优于现有基线方法，相比先前的最优方法平均性能提升8.4%，证明了其在长视频推理上的有效性。