Video anomaly detection (VAD) with weak supervision has achieved remarkable performance in utilizing video-level labels to discriminate whether a video frame is normal or abnormal. However, current approaches are inherently limited to a closed-set setting and may struggle in open-world applications where there can be anomaly categories in the test data unseen during training. A few recent studies attempt to tackle a more realistic setting, open-set VAD, which aims to detect unseen anomalies given seen anomalies and normal videos. However, such a setting focuses on predicting frame anomaly scores, having no ability to recognize the specific categories of anomalies, despite the fact that this ability is essential for building more informed video surveillance systems. This paper takes a step further and explores open-vocabulary video anomaly detection (OVVAD), in which we aim to leverage pre-trained large models to detect and categorize seen and unseen anomalies. To this end, we propose a model that decouples OVVAD into two mutually complementary tasks -- class-agnostic detection and class-specific classification -- and jointly optimizes both tasks. Particularly, we devise a semantic knowledge injection module to introduce semantic knowledge from large language models for the detection task, and design a novel anomaly synthesis module to generate pseudo unseen anomaly videos with the help of large vision generation models for the classification task. These semantic knowledge and synthesis anomalies substantially extend our model's capability in detecting and categorizing a variety of seen and unseen anomalies. Extensive experiments on three widely-used benchmarks demonstrate our model achieves state-of-the-art performance on OVVAD task.

翻译：视频异常检测（VAD）在利用视频级标签区分帧级正常与异常行为方面，凭借弱监督方法已取得显著性能。然而，现有方法本质上局限于封闭集设定，在开放世界应用中可能难以应对测试数据中出现训练时未见过的异常类别。近期部分研究试图处理更现实的开放集VAD设定，即通过已知异常和正常视频检测未知异常。但该类设定仅聚焦于预测帧级异常分数，无法识别异常的具体类别——尽管该能力对构建更完善的视频监控系统至关重要。本文进一步探索开放式词汇视频异常检测（OVVAD），旨在利用预训练大模型检测并分类已知与未知异常。为此，我们提出将OVVAD解耦为两个互补任务（类别无关检测与类别特异性分类）并联合优化的模型。具体而言，我们设计语义知识注入模块从大语言模型中引入语义知识辅助检测任务，并创新性提出异常合成模块，借助大型视觉生成模型为分类任务生成伪未知异常视频。这些语义知识与合成异常显著提升了模型检测与分类各类已知/未知异常的能力。在三个广泛使用的基准上的大量实验表明，本模型在OVVAD任务上达到了最优性能。