Temporal video grounding (TVG) is a critical task in video content understanding. Despite significant advancements, existing methods often limit in capturing the fine-grained relationships between multimodal inputs and the high computational costs with processing long video sequences. To address these limitations, we introduce a novel SpikeMba: multi-modal spiking saliency mamba for temporal video grounding. In our work, we integrate the Spiking Neural Networks (SNNs) and state space models (SSMs) to capture the fine-grained relationships of multimodal features effectively. Specifically, we introduce the relevant slots to enhance the model's memory capabilities, enabling a deeper contextual understanding of video sequences. The contextual moment reasoner leverages these slots to maintain a balance between contextual information preservation and semantic relevance exploration. Simultaneously, the spiking saliency detector capitalizes on the unique properties of SNNs to accurately locate salient proposals. Our experiments demonstrate the effectiveness of SpikeMba, which consistently outperforms state-of-the-art methods across mainstream benchmarks.

翻译：时序视频定位（TVG）是视频内容理解中的关键任务。尽管已有显著进展，现有方法在捕获多模态输入间的细粒度关系以及处理长视频序列时的高计算开销方面仍存在局限。为解决这些不足，我们提出了一种新型模型SpikeMba：面向时序视频定位的多模态脉冲显著性曼巴。在本工作中，我们融合了脉冲神经网络（SNNs）与状态空间模型（SSMs），以有效捕获多模态特征的细粒度关系。具体而言，我们引入相关槽（relevant slots）以增强模型的记忆能力，从而实现对视频序列更深入的上下文理解。上下文时刻推理器利用这些槽在上下文信息保留与语义相关性探索之间维持平衡。同时，脉冲显著性检测器利用SNNs的独特特性来准确定位显著性候选区域。实验证明了SpikeMba的有效性，其在主流基准测试中持续优于当前最先进方法。