Video-text Large Language Models (video-text LLMs) have shown remarkable performance in answering questions and holding conversations on simple videos. However, they perform almost the same as random on grounding text queries in long and complicated videos, having little ability to understand and reason about temporal information, which is the most fundamental difference between videos and images. In this paper, we propose HawkEye, one of the first video-text LLMs that can perform temporal video grounding in a fully text-to-text manner. To collect training data that is applicable for temporal video grounding, we construct InternVid-G, a large-scale video-text corpus with segment-level captions and negative spans, with which we introduce two new time-aware training objectives to video-text LLMs. We also propose a coarse-grained method of representing segments in videos, which is more robust and easier for LLMs to learn and follow than other alternatives. Extensive experiments show that HawkEye is better at temporal video grounding and comparable on other video-text tasks with existing video-text LLMs, which verifies its superior video-text multi-modal understanding abilities.

翻译：摘要：视频-文本大语言模型在回答简单视频问题及进行对话方面表现出色。然而，在处理长视频中复杂的文本查询定位任务时，其表现近乎随机，对时间信息（视频与图像最本质的区别）的理解与推理能力极为有限。本文提出HawkEye——首批能以全文本到文本方式实现时间视频定位的视频-文本大语言模型之一。为收集适用于时间视频定位的训练数据，我们构建了InternVid-G数据集——一个包含片段级标注及负向时间跨段的大规模视频-文本语料库，并据此引入了两个面向视频-文本大语言模型的时间感知训练目标。我们还提出了一种粗粒度视频片段表示方法，相较于其他方案，该方法更具鲁棒性且更易于大语言模型学习与遵循。大量实验表明，HawkEye在时间视频定位任务上表现更优，在其他视频-文本任务中与现有模型性能相当，验证了其卓越的视频-文本多模态理解能力。