Open-world video recognition is challenging since traditional networks are not generalized well on complex environment variations. Alternatively, foundation models with rich knowledge have recently shown their generalization power. However, how to apply such knowledge has not been fully explored for open-world video recognition. To this end, we propose a generic knowledge transfer pipeline, which progressively exploits and integrates external multimodal knowledge from foundation models to boost open-world video recognition. We name it PCA, based on three stages of Percept, Chat, and Adapt. First, we perform Percept process to reduce the video domain gap and obtain external visual knowledge. Second, we generate rich linguistic semantics as external textual knowledge in Chat stage. Finally, we blend external multimodal knowledge in Adapt stage, by inserting multimodal knowledge adaptation modules into networks. We conduct extensive experiments on three challenging open-world video benchmarks, i.e., TinyVIRAT, ARID, and QV-Pipe. Our approach achieves state-of-the-art performance on all three datasets.

翻译：开放世界视频识别具有挑战性，因为传统网络在复杂环境变化下泛化能力有限。近期，具备丰富知识的基础模型展现了强大的泛化能力。然而，如何将此类知识应用于开放世界视频识别尚未得到充分探索。为此，我们提出一种通用的知识迁移流程，逐步利用并整合来自基础模型的外部多模态知识，以提升开放世界视频识别性能。我们将其命名为PCA，基于感知、对话和适应三个阶段。首先，通过感知过程减少视频领域差异并获取外部视觉知识。其次，在对话阶段生成丰富的语言语义作为外部文本知识。最后，在适应阶段通过向网络中插入多模态知识适应模块，融合外部多模态知识。我们在三个具有挑战性的开放世界视频基准数据集（即TinyVIRAT、ARID和QV-Pipe）上进行了广泛实验。我们的方法在所有三个数据集上均取得了最先进的性能。