We present the Object Language Video Transformer (OLViT) - a novel model for video dialog operating over a multi-modal attention-based dialog state tracker. Existing video dialog models struggle with questions requiring both spatial and temporal localization within videos, long-term temporal reasoning, and accurate object tracking across multiple dialog turns. OLViT addresses these challenges by maintaining a global dialog state based on the output of an Object State Tracker (OST) and a Language State Tracker (LST): while the OST attends to the most important objects within the video, the LST keeps track of the most important linguistic co-references to previous dialog turns. In stark contrast to previous works, our approach is generic by nature and is therefore capable of learning continuous multi-modal dialog state representations of the most relevant objects and rounds. As a result, they can be seamlessly integrated into Large Language Models (LLMs) and offer high flexibility in dealing with different datasets and tasks. Evaluations on the challenging DVD (response classification) and SIMMC 2.1 (response generation) datasets show that OLViT achieves new state-of-the-art performance across both datasets.

翻译：我们提出对象语言视频Transformer（OLViT）——一种新颖的视频对话模型，该模型基于多模态注意力对话状态追踪器运行。现有视频对话模型难以应对需要视频内空间与时间定位、长期时间推理以及在多个对话轮次中精确追踪对象的任务。OLViT通过维护基于对象状态追踪器（OST）和语言状态追踪器（LST）输出的全局对话状态来解决这些挑战：OST关注视频中最重要的对象，而LST追踪先前对话轮次中最重要的语言共指关系。与以往工作截然不同的是，我们的方法本质上具有通用性，因此能够学习最相关对象和轮次的连续多模态对话状态表示。因此，它们可以无缝集成到大型语言模型（LLM）中，并在处理不同数据集和任务时展现出高度灵活性。在具有挑战性的DVD（响应分类）和SIMMC 2.1（响应生成）数据集上的评估表明，OLViT在两个数据集上均实现了新的最优性能。