We introduce TemporalVLM, a video large language model (video LLM) for temporal reasoning and fine-grained understanding in long videos. Our approach includes a visual encoder for mapping a long-term video into features which are time-aware and contain both local and global cues. It first divides an input video into short-term clips, which are jointly encoded with timestamps and fused across overlapping temporal windows into time-sensitive local features. Next, the local features are passed through a bidirectional long short-term memory (BiLSTM) module for global feature aggregation. Moreover, to facilitate the evaluation of TemporalVLM, we present a large-scale long video dataset of industry assembly processes, namely IndustryASM, consisting of videos recorded on factory floors with actions and timestamps annotated by industrial engineers for time and motion studies and temporal action segmentation evaluation. Finally, extensive experiments show that TemporalVLM outperforms previous methods across temporal reasoning and fine-grained understanding tasks, i.e., dense video captioning, temporal video grounding, video highlight detection, and temporal action segmentation. To our best knowledge, our work is the first to incorporate LSTMs into video LLMs.

翻译：我们提出了TemporalVLM，一种用于长视频时序推理与细粒度理解的视频大语言模型。我们的方法包含一个视觉编码器，用于将长时视频映射为具有时间感知且包含局部与全局线索的特征。该方法首先将输入视频划分为短期片段，这些片段与时间戳联合编码，并通过重叠时序窗口融合为时间敏感的局部特征。随后，局部特征通过一个双向长短期记忆模块进行全局特征聚合。此外，为便于评估TemporalVLM，我们提出了一个大规模工业装配过程长视频数据集，即IndustryASM。该数据集包含在工厂车间录制的视频，其动作与时间戳由工业工程师标注，可用于时间与动作研究以及时序动作分割评估。最后，大量实验表明，TemporalVLM在时序推理与细粒度理解任务（即密集视频描述、时序视频定位、视频高光检测和时序动作分割）上均优于先前方法。据我们所知，我们的工作是首次将LSTM融入视频大语言模型。