The recent contrastive language-image pre-training (CLIP) model has shown great success in a wide range of image-level tasks, revealing remarkable ability for learning powerful visual representations with rich semantics. An open and worthwhile problem is efficiently adapting such a strong model to the video domain and designing a robust video anomaly detector. In this work, we propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) by leveraging the frozen CLIP model directly without any pre-training and fine-tuning process. Unlike current works that directly feed extracted features into the weakly supervised classifier for frame-level binary classification, VadCLIP makes full use of fine-grained associations between vision and language on the strength of CLIP and involves dual branch. One branch simply utilizes visual features for coarse-grained binary classification, while the other fully leverages the fine-grained language-image alignment. With the benefit of dual branch, VadCLIP achieves both coarse-grained and fine-grained video anomaly detection by transferring pre-trained knowledge from CLIP to WSVAD task. We conduct extensive experiments on two commonly-used benchmarks, demonstrating that VadCLIP achieves the best performance on both coarse-grained and fine-grained WSVAD, surpassing the state-of-the-art methods by a large margin. Specifically, VadCLIP achieves 84.51% AP and 88.02% AUC on XD-Violence and UCF-Crime, respectively. Code and features will be released to facilitate future VAD research.

翻译：近期提出的对比语言-图像预训练（CLIP）模型在广泛图像级任务中展现出卓越性能，揭示了其学习富含语义的强视觉表征的非凡能力。如何高效地将这一强大模型适配至视频域并构建鲁棒的视频异常检测器，是一个开放且值得探索的问题。本文提出VadCLIP——一种面向弱监督视频异常检测（WSVAD）的新范式，该方法直接利用冻结的CLIP模型，无需任何预训练或微调过程。不同于现有方法将提取的特征直接输入弱监督分类器进行帧级二分类，VadCLIP充分发挥CLIP在视觉与语言间细粒度关联上的优势，并引入双分支结构：一个分支仅利用视觉特征进行粗粒度二分类，另一分支则充分利用细粒度语言-图像对齐能力。凭借双分支的优势，VadCLIP通过将CLIP的预训练知识迁移至WSVAD任务，实现了粗粒度与细粒度的视频异常检测。在两个常用基准数据集上的大量实验表明，VadCLIP在粗粒度和细粒度WSVAD任务中均取得最优性能，并以显著优势超越现有最先进方法。具体而言，VadCLIP在XD-Violence和UCF-Crime数据集上分别达到84.51%的AP和88.02%的AUC。代码与特征将公开发布以促进未来VAD研究。