We introduce V-Agent, a novel multi-agent platform designed for advanced video search and interactive user-system conversations. By fine-tuning a vision-language model (VLM) with a small video preference dataset and enhancing it with a retrieval vector from an image-text retrieval model, we overcome the limitations of traditional text-based retrieval systems in multimodal scenarios. The VLM-based retrieval model independently embeds video frames and audio transcriptions from an automatic speech recognition (ASR) module into a shared multimodal representation space, enabling V-Agent to interpret both visual and spoken content for context-aware video search. This system consists of three agents-a routing agent, a search agent, and a chat agent-that work collaboratively to address user intents by refining search outputs and communicating with users. The search agent utilizes the VLM-based retrieval model together with an additional re-ranking module to further enhance video retrieval quality. Our proposed framework demonstrates state-of-the-art zero-shot performance on the MultiVENT 2.0 benchmark, highlighting its potential for both academic research and real-world applications. The retrieval model and demo videos are available at https://huggingface.co/NCSOFT/multimodal-embedding.

翻译：本文介绍V-Agent——一个为高级视频搜索与交互式用户-系统对话设计的新型多智能体平台。通过使用小型视频偏好数据集对视觉语言模型进行微调，并借助图像-文本检索模型的检索向量进行增强，我们克服了传统文本检索系统在多模态场景中的局限性。基于VLM的检索模型将来自自动语音识别模块的视频帧和音频转录文本独立嵌入到共享的多模态表示空间中，使V-Agent能够同时解析视觉与语音内容，实现情境感知的视频搜索。该系统包含路由智能体、搜索智能体和对话智能体三个组件，通过优化搜索结果并与用户持续交互来协同响应用户意图。搜索智能体结合基于VLM的检索模型与额外的重排序模块，进一步提升视频检索质量。我们在MultiVENT 2.0基准测试中实现了先进的零样本性能，彰显了该框架在学术研究与实际应用中的潜力。检索模型与演示视频已发布于https://huggingface.co/NCSOFT/multimodal-embedding。