Audio-visual large language models (LLMs) hold strong promise for long-form video understanding, yet their long-video inference is fundamentally limited by the linear growth of video tokens and key-value (KV) caches. We present OmniMem, a memory-efficient streaming framework designed specifically for audio-visual LLMs. Unlike existing compression methods that treat all tokens uniformly, OmniMem introduces a modality-aware memory allocation strategy that separately manages visual and audio contexts, addressing the severe token imbalance between the two modalities. OmniMem further preserves informative and non-redundant KV states through perturbation-aware memory selection, enabling compact memory without sacrificing long-range understanding. To strengthen compression under realistic deployment constraints, we also explore budget-aware fine-tuning, which encourages the model to consolidate useful information into retained memory. Experiments on VideoMME Long, LVBench, and LVOmniBench with video-SALMONN 2+ and Qwen-2.5-Omni show that OmniMem consistently improves over strong training-free compression baselines by 2-4% absolute accuracy under the same memory budgets, with an additional 1-2% gain after fine-tuning.

翻译：音视频大语言模型（LLMs）在长视频理解方面展现出巨大潜力，但其长视频推理能力从根本上受限于视频令牌和键值（KV）缓存的线性增长。我们提出OmniMem，一种专为音视频LLMs设计的高效记忆流式推理框架。与现有对所有令牌统一处理的压缩方法不同，OmniMem引入模态感知的记忆分配策略，分别管理视觉和音频上下文，解决两种模态间严重的令牌不平衡问题。OmniMem进一步通过扰动感知的记忆选择保留信息量丰富且无冗余的KV状态，在实现紧凑记忆的同时不牺牲长程理解能力。为增强实际部署约束下的压缩效果，我们还探索了预算感知的微调方法，促使模型将有用信息整合至保留的记忆中。在VideoMME Long、LVBench和LVOmniBench上，基于video-SALMONN 2+和Qwen-2.5-Omni的实验表明，OmniMem在相同记忆预算下始终较强大的无训练压缩基线方法提升2-4%的绝对准确率，微调后还可额外获得1-2%的提升。