Pre-trained vision-language models (VLMs) have enabled significant progress in open vocabulary computer vision tasks such as image classification, object detection and image segmentation. Some recent works have focused on extending VLMs to open vocabulary single label action classification in videos. However, previous methods fall short in holistic video understanding which requires the ability to simultaneously recognize multiple actions and entities e.g., objects in the video in an open vocabulary setting. We formulate this problem as open vocabulary multilabel video classification and propose a method to adapt a pre-trained VLM such as CLIP to solve this task. We leverage large language models (LLMs) to provide semantic guidance to the VLM about class labels to improve its open vocabulary performance with two key contributions. First, we propose an end-to-end trainable architecture that learns to prompt an LLM to generate soft attributes for the CLIP text-encoder to enable it to recognize novel classes. Second, we integrate a temporal modeling module into CLIP's vision encoder to effectively model the spatio-temporal dynamics of video concepts as well as propose a novel regularized finetuning technique to ensure strong open vocabulary classification performance in the video domain. Our extensive experimentation showcases the efficacy of our approach on multiple benchmark datasets.

翻译：预训练的视觉-语言模型（VLMs）在开放词汇计算机视觉任务（如图像分类、目标检测和图像分割）中取得了显著进展。近期的一些研究专注于将VLMs扩展至视频中的开放词汇单标签动作分类。然而，先前的方法在整体视频理解方面存在不足，这需要模型能够在开放词汇设置下同时识别视频中的多个动作和实体（例如物体）。我们将此问题形式化为开放词汇多标签视频分类，并提出一种方法来调整预训练的VLM（如CLIP）以解决此任务。我们利用大语言模型（LLMs）为VLM提供关于类别标签的语义指导，以通过两个关键贡献提升其开放词汇性能。首先，我们提出一种端到端可训练的架构，该架构学习提示LLM为CLIP文本编码器生成软属性，使其能够识别新类别。其次，我们将一个时序建模模块集成到CLIP的视觉编码器中，以有效建模视频概念的时空动态，并提出一种新颖的正则化微调技术，以确保在视频领域具有强大的开放词汇分类性能。我们广泛的实验展示了该方法在多个基准数据集上的有效性。