Action knowledge involves the understanding of textual, visual, and temporal aspects of actions. We introduce the Action Dynamics Benchmark (ActionBench) containing two carefully designed probing tasks: Action Antonym and Video Reversal, which targets multimodal alignment capabilities and temporal understanding skills of the model, respectively. Despite recent video-language models' (VidLM) impressive performance on various benchmark tasks, our diagnostic tasks reveal their surprising deficiency (near-random performance) in action knowledge, suggesting that current models rely on object recognition abilities as a shortcut for action understanding. To remedy this, we propose a novel framework, Paxion, along with a new Discriminative Video Dynamics Modeling (DVDM) objective. The Paxion framework utilizes a Knowledge Patcher network to encode new action knowledge and a Knowledge Fuser component to integrate the Patcher into frozen VidLMs without compromising their existing capabilities. Due to limitations of the widely-used Video-Text Contrastive (VTC) loss for learning action knowledge, we introduce the DVDM objective to train the Knowledge Patcher. DVDM forces the model to encode the correlation between the action text and the correct ordering of video frames. Our extensive analyses show that Paxion and DVDM together effectively fill the gap in action knowledge understanding (~50% to 80%), while maintaining or improving performance on a wide spectrum of both object- and action-centric downstream tasks.

翻译：动作知识涉及对动作的文本、视觉和时间维度的理解。我们提出了动作动态基准（ActionBench），包含两个精心设计的探查任务：动作反义词与视频反转，分别针对模型的多模态对齐能力和时间理解技能。尽管近期视频-语言模型（VidLM）在各种基准任务上表现卓越，但我们的诊断任务揭示了其在动作知识方面存在惊人的缺陷（接近随机水平的性能），表明当前模型依赖物体识别能力作为理解动作的捷径。为解决这一问题，我们提出新型框架Paxion，并引入新的判别式视频动态建模（DVDM）目标。Paxion框架利用知识修补网络编码新动作知识，并通过知识融合组件将修补器集成到冻结的VidLM中，同时不损害其现有能力。针对广泛用于学习动作知识的视频-文本对比（VTC）损失的局限性，我们提出DVDM目标来训练知识修补器。DVDM强制模型编码动作文本与视频帧正确排序之间的相关性。广泛的分析表明，Paxion与DVDM共同有效填补了动作知识理解中的缺口（从约50%提升至80%），同时在面向物体和动作的广泛下游任务中保持或提升了性能。