Video anomaly detection (VAD) with weak supervision has achieved remarkable performance in utilizing video-level labels to discriminate whether a video frame is normal or abnormal. However, current approaches are inherently limited to a closed-set setting and may struggle in open-world applications where there can be anomaly categories in the test data unseen during training. A few recent studies attempt to tackle a more realistic setting, open-set VAD, which aims to detect unseen anomalies given seen anomalies and normal videos. However, such a setting focuses on predicting frame anomaly scores, having no ability to recognize the specific categories of anomalies, despite the fact that this ability is essential for building more informed video surveillance systems. This paper takes a step further and explores open-vocabulary video anomaly detection (OVVAD), in which we aim to leverage pre-trained large models to detect and categorize seen and unseen anomalies. To this end, we propose a model that decouples OVVAD into two mutually complementary tasks -- class-agnostic detection and class-specific classification -- and jointly optimizes both tasks. Particularly, we devise a semantic knowledge injection module to introduce semantic knowledge from large language models for the detection task, and design a novel anomaly synthesis module to generate pseudo unseen anomaly videos with the help of large vision generation models for the classification task. These semantic knowledge and synthesis anomalies substantially extend our model's capability in detecting and categorizing a variety of seen and unseen anomalies. Extensive experiments on three widely-used benchmarks demonstrate our model achieves state-of-the-art performance on OVVAD task.

翻译：弱监督视频异常检测（VAD）在利用视频级标签判别帧级正常/异常方面取得了显著性能。然而，现有方法固有地局限于闭集设定，难以应对开放世界应用中训练数据未涵盖的异常类别。近期研究尝试探索更现实的开集VAD设定，旨在利用已知异常和正常视频检测未知异常。但这种设定仅预测帧异常分数，无法识别具体异常类别——尽管这一能力对构建更完善的视频监控系统至关重要。本文进一步探索开放词汇视频异常检测（OVVAD），旨在利用预训练大模型检测并分类已知与未知异常。为此，我们提出将OVVAD解耦为两个互补任务——类别无关检测与类别特定分类——并联合优化两项任务。具体而言，我们设计语义知识注入模块，从大语言模型引入语义知识辅助检测任务；并设计新颖的异常合成模块，借助大视觉生成模型为分类任务生成伪未见异常视频。这些语义知识与合成异常显著扩展了模型检测和分类各类已知/未知异常的能力。在三个广泛使用的基准上的大量实验表明，我们的模型在OVVAD任务上达到了最先进性能。