Enabling large language models (LLMs) to read videos is vital for multimodal LLMs. Existing works show promise on short videos whereas long video (longer than e.g.~1 minute) comprehension remains challenging. The major problem lies in the over-compression of videos, i.e., the encoded video representations are not enough to represent the whole video. To address this issue, we propose Long Video Chat (LVChat), where Frame-Scalable Encoding (FSE) is introduced to dynamically adjust the number of embeddings in alignment with the duration of the video to ensure long videos are not overly compressed into a few embeddings. To deal with long videos whose length is beyond videos seen during training, we propose Interleaved Frame Encoding (IFE), repeating positional embedding and interleaving multiple groups of videos to enable long video input, avoiding performance degradation due to overly long videos. Experimental results show that LVChat significantly outperforms existing methods by up to 27\% in accuracy on long-video QA datasets and long-video captioning benchmarks. Our code is published at https://github.com/wangyu-ustc/LVChat.

翻译：使大型语言模型（LLMs）能够读取视频对于多模态LLMs至关重要。现有研究在短视频上表现出潜力，但长视频（例如超过1分钟）的理解仍具挑战性。主要问题在于视频的过度压缩，即编码后的视频表示不足以代表完整视频。为解决此问题，我们提出长视频对话模型（LVChat），其中引入帧可扩展编码（FSE）以动态调整嵌入数量，使其与视频时长对齐，从而确保长视频不会被过度压缩为少量嵌入。为处理长度超出训练视频范围的长视频，我们提出交错帧编码（IFE），通过重复位置嵌入并交错多组视频来支持长视频输入，避免因视频过长导致的性能下降。实验结果表明，LVChat在长视频问答数据集和长视频字幕基准测试中，准确率相较于现有方法最高提升27%。我们的代码已发布于 https://github.com/wangyu-ustc/LVChat。