This paper introduces QCaption, a novel video captioning and Q&A pipeline that enhances video analytics by fusing three models: key frame extraction, a Large Multimodal Model (LMM) for image-text analysis, and a Large Language Model (LLM) for text analysis. This approach enables integrated analysis of text, images, and video, achieving performance improvements over existing video captioning and Q&A models; all while remaining fully self-contained, adept for on-premises deployment. Experimental results using QCaption demonstrated up to 44.2% and 48.9% improvements in video captioning and Q&A tasks, respectively. Ablation studies were also performed to assess the role of LLM on the fusion on the results. Moreover, the paper proposes and evaluates additional video captioning approaches, benchmarking them against QCaption and existing methodologies. QCaption demonstrate the potential of adopting a model fusion approach in advancing video analytics.

翻译：本文提出QCaption，一种新颖的视频描述生成与问答流程，通过融合三个模型来增强视频分析能力：关键帧提取模块、用于图文分析的大型多模态模型（LMM）以及用于文本分析的大型语言模型（LLM）。该方法实现了文本、图像与视频的集成分析，在保持完全自包含且适配本地化部署的同时，其性能超越了现有视频描述生成与问答模型。使用QCaption的实验结果表明，在视频描述生成和问答任务上分别实现了最高44.2%和48.9%的性能提升。本文还通过消融实验评估了LLM在模型融合中对结果的影响。此外，研究提出并评估了其他视频描述生成方法，将其与QCaption及现有方法进行基准比较。QCaption证明了采用模型融合方法在推进视频分析领域具有重要潜力。