The recent contrastive language-image pre-training (CLIP) model has shown great success in a wide range of image-level tasks, revealing remarkable ability for learning powerful visual representations with rich semantics. An open and worthwhile problem is efficiently adapting such a strong model to the video domain and designing a robust video anomaly detector. In this work, we propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) by leveraging the frozen CLIP model directly without any pre-training and fine-tuning process. Unlike current works that directly feed extracted features into the weakly supervised classifier for frame-level binary classification, VadCLIP makes full use of fine-grained associations between vision and language on the strength of CLIP and involves dual branch. One branch simply utilizes visual features for coarse-grained binary classification, while the other fully leverages the fine-grained language-image alignment. With the benefit of dual branch, VadCLIP achieves both coarse-grained and fine-grained video anomaly detection by transferring pre-trained knowledge from CLIP to WSVAD task. We conduct extensive experiments on two commonly-used benchmarks, demonstrating that VadCLIP achieves the best performance on both coarse-grained and fine-grained WSVAD, surpassing the state-of-the-art methods by a large margin. Specifically, VadCLIP achieves 84.51% AP and 88.02% AUC on XD-Violence and UCF-Crime, respectively. Code and features will be released to facilitate future VAD research.

翻译：最近的对比语言-图像预训练（CLIP）模型在一系列图像级任务中取得了巨大成功，展现出学习具有丰富语义的强视觉表示能力的卓越性能。一个开放且值得探讨的问题是如何有效地将如此强大的模型适应到视频领域，并设计鲁棒的视频异常检测器。本文提出VadCLIP，一种利用冻结的CLIP模型直接进行弱监督视频异常检测（WSVAD）的新范式，无需任何预训练和微调过程。与当前直接将提取的特征输入弱监督分类器进行帧级二分类的工作不同，VadCLIP借助CLIP的优势充分利用视觉与语言之间的细粒度关联，并引入双分支架构。一个分支仅利用视觉特征进行粗粒度二分类，而另一个分支则充分利用细粒度的语言-图像对齐。借助双分支的优势，VadCLIP通过将CLIP的预训练知识迁移到WSVAD任务，实现了粗粒度和细粒度的视频异常检测。我们在两个常用基准数据集上进行了大量实验，结果表明VadCLIP在粗粒度和细粒度WSVAD上均达到了最佳性能，大幅超越了现有最先进方法。具体而言，VadCLIP在XD-Violence和UCF-Crime数据集上分别实现了84.51%的AP和88.02%的AUC。代码和特征将公开发布，以促进未来VAD研究。