The recent contrastive language-image pre-training (CLIP) model has shown great success in a wide range of image-level tasks, revealing remarkable ability for learning powerful visual representations with rich semantics. An open and worthwhile problem is efficiently adapting such a strong model to the video domain and designing a robust video anomaly detector. In this work, we propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) by leveraging the frozen CLIP model directly without any pre-training and fine-tuning process. Unlike current works that directly feed extracted features into the weakly supervised classifier for frame-level binary classification, VadCLIP makes full use of fine-grained associations between vision and language on the strength of CLIP and involves dual branch. One branch simply utilizes visual features for coarse-grained binary classification, while the other fully leverages the fine-grained language-image alignment. With the benefit of dual branch, VadCLIP achieves both coarse-grained and fine-grained video anomaly detection by transferring pre-trained knowledge from CLIP to WSVAD task. We conduct extensive experiments on two commonly-used benchmarks, demonstrating that VadCLIP achieves the best performance on both coarse-grained and fine-grained WSVAD, surpassing the state-of-the-art methods by a large margin. Specifically, VadCLIP achieves 84.51% AP and 88.02% AUC on XD-Violence and UCF-Crime, respectively. Code and features are released at https://github.com/nwpu-zxr/VadCLIP.

翻译：近期提出的对比语言-图像预训练（CLIP）模型在各类图像级任务中展现了卓越性能，揭示了其学习富含语义的强视觉表征的非凡能力。如何高效地将这一强大模型适配至视频领域并构建鲁棒的视频异常检测器，是一个开放且值得探究的问题。本文提出VadCLIP——一种面向弱监督视频异常检测（WSVAD）的新范式，该方法无需任何预训练或微调过程，直接利用冻结的CLIP模型。不同于现有工作直接将提取的特征输入弱监督分类器进行帧级二分类，VadCLIP借助CLIP的优势充分利用视觉与语言之间的细粒度关联，并引入双分支结构。一个分支仅使用视觉特征进行粗粒度二分类，另一分支则充分利用细粒度语言-图像对齐能力。得益于双分支设计，VadCLIP通过将CLIP的预训练知识迁移至WSVAD任务，实现了粗粒度与细粒度视频异常检测。在两个通用基准上开展的广泛实验表明，VadCLIP在粗粒度与细粒度WSVAD任务上均达到最优性能，显著超越现有方法。具体而言，VadCLIP在XD-Violence和UCF-Crime数据集上分别取得84.51%的AP与88.02%的AUC。代码与特征已发布于https://github.com/nwpu-zxr/VadCLIP。