Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain. The proposed VideoMamba overcomes the limitations of existing 3D convolution neural networks and video transformers. Its linear-complexity operator enables efficient long-term modeling, which is crucial for high-resolution long video understanding. Extensive evaluations reveal VideoMamba's four core abilities: (1) Scalability in the visual domain without extensive dataset pretraining, thanks to a novel self-distillation technique; (2) Sensitivity for recognizing short-term actions even with fine-grained motion differences; (3) Superiority in long-term video understanding, showcasing significant advancements over traditional feature-based models; and (4) Compatibility with other modalities, demonstrating robustness in multi-modal contexts. Through these distinct advantages, VideoMamba sets a new benchmark for video understanding, offering a scalable and efficient solution for comprehensive video understanding. All the code and models are available at https://github.com/OpenGVLab/VideoMamba.

翻译：针对视频理解中局部冗余与全局依赖的双重挑战，本文创新性地将Mamba模型适配至视频领域。所提出的VideoMamba克服了现有3D卷积神经网络与视频Transformer的局限性，其线性复杂度算子可实现高效的长时建模，这对高分辨率长视频理解至关重要。广泛评估揭示了VideoMamba的四大核心能力：（1）通过新颖的自蒸馏技术，无需大规模数据集预训练即可在视觉领域实现可扩展性；（2）即使面对细微运动差异，仍具备识别短期动作的敏感性；（3）在长时视频理解方面展现超越传统基于特征模型的显著优势；（4）与其他模态的兼容性，在多模态场景中表现出鲁棒性。凭借这些独特优势，VideoMamba为视频理解树立了新标杆，提供了可扩展且高效的综合性视频理解方案。所有代码与模型均已开源：https://github.com/OpenGVLab/VideoMamba。