Video anomaly detection (VAD) with weak supervision has achieved remarkable performance in utilizing video-level labels to discriminate whether a video frame is normal or abnormal. However, current approaches are inherently limited to a closed-set setting and may struggle in open-world applications where there can be anomaly categories in the test data unseen during training. A few recent studies attempt to tackle a more realistic setting, open-set VAD, which aims to detect unseen anomalies given seen anomalies and normal videos. However, such a setting focuses on predicting frame anomaly scores, having no ability to recognize the specific categories of anomalies, despite the fact that this ability is essential for building more informed video surveillance systems. This paper takes a step further and explores open-vocabulary video anomaly detection (OVVAD), in which we aim to leverage pre-trained large models to detect and categorize seen and unseen anomalies. To this end, we propose a model that decouples OVVAD into two mutually complementary tasks -- class-agnostic detection and class-specific classification -- and jointly optimizes both tasks. Particularly, we devise a semantic knowledge injection module to introduce semantic knowledge from large language models for the detection task, and design a novel anomaly synthesis module to generate pseudo unseen anomaly videos with the help of large vision generation models for the classification task. These semantic knowledge and synthesis anomalies substantially extend our model's capability in detecting and categorizing a variety of seen and unseen anomalies. Extensive experiments on three widely-used benchmarks demonstrate our model achieves state-of-the-art performance on OVVAD task.

翻译：弱监督下的视频异常检测（VAD）在利用视频级标签判定帧是否正常方面取得了显著成效。然而，现有方法本质上局限于封闭集场景，在开放世界应用中可能面临挑战，因为测试数据中可能存在训练时未见过的异常类别。近期少数研究尝试应对更现实的开放集VAD场景，旨在通过已知异常与正常视频检测未知异常。但这种设定仅侧重于预测帧异常分数，无法识别具体异常类别——尽管该能力对构建更完善的视频监控系统至关重要。本文进一步探索开放词汇视频异常检测（OVVAD），旨在利用预训练大模型检测并分类已知与未知异常。为此，我们提出一种解耦OVVAD为两个互补任务——类别无关检测与类别特定分类——并联合优化的模型。具体而言，我们设计语义知识注入模块将大语言模型的语义知识引入检测任务，并创新性提出异常合成模块，借助大型视觉生成模型为分类任务生成伪未知异常视频。这些语义知识与合成异常显著扩展了模型检测与分类各类已知/未知异常的能力。在三个广泛使用的基准上的大量实验表明，本模型在OVVAD任务上达到了最优性能。