Text-video retrieval, a prominent sub-field within the domain of multimodal information retrieval, has witnessed remarkable growth in recent years. However, existing methods assume video scenes are consistent with unbiased descriptions. These limitations fail to align with real-world scenarios since descriptions can be influenced by annotator biases, diverse writing styles, and varying textual perspectives. To overcome the aforementioned problems, we introduce $\texttt{WAVER}$, a cross-domain knowledge distillation framework via vision-language models through open-vocabulary knowledge designed to tackle the challenge of handling different writing styles in video descriptions. $\texttt{WAVER}$ capitalizes on the open-vocabulary properties that lie in pre-trained vision-language models and employs an implicit knowledge distillation approach to transfer text-based knowledge from a teacher model to a vision-based student. Empirical studies conducted across four standard benchmark datasets, encompassing various settings, provide compelling evidence that $\texttt{WAVER}$ can achieve state-of-the-art performance in text-video retrieval task while handling writing-style variations. The code is available at: https://github.com/Fsoft-AIC/WAVER
翻译:文本-视频检索作为多模态信息检索领域中的关键子方向,近年来取得了显著发展。然而现有方法假设视频场景与无偏描述保持一致,这一局限性难以适配真实场景——因为描述可能受到标注者偏好、多样写作风格及差异化文本视角的影响。为克服上述问题,我们提出$\texttt{WAVER}$——一种基于视觉语言模型通过开放词汇知识进行跨域知识蒸馏的框架,专门应对视频描述中不同写作风格带来的挑战。$\texttt{WAVER}$利用预训练视觉语言模型的开放词汇特性,采用隐式知识蒸馏方法将教师模型中的文本知识迁移至基于视觉的学生模型。在涵盖多种场景的四个标准基准数据集上的实证研究表明,$\texttt{WAVER}$在处理写作风格差异的同时,能在文本-视频检索任务中取得最先进的性能。代码已开源:https://github.com/Fsoft-AIC/WAVER