We propose a new task and model for dense video object captioning -- detecting, tracking, and captioning trajectories of all objects in a video. This task unifies spatial and temporal understanding of the video, and requires fine-grained language description. Our model for dense video object captioning is trained end-to-end and consists of different modules for spatial localization, tracking, and captioning. As such, we can train our model with a mixture of disjoint tasks, and leverage diverse, large-scale datasets which supervise different parts of our model. This results in noteworthy zero-shot performance. Moreover, by finetuning a model from this initialization, we can further improve our performance, surpassing strong image-based baselines by a significant margin. Although we are not aware of other work performing this task, we are able to repurpose existing video grounding datasets for our task, namely VidSTG and VLN. We show our task is more general than grounding, and models trained on our task can directly be applied to grounding by finding the bounding box with the maximum likelihood of generating the query sentence. Our model outperforms dedicated, state-of-the-art models for spatial grounding on both VidSTG and VLN.

翻译：我们提出了一项新任务及其模型，用于密集视频对象描述——即对视频中所有对象的轨迹进行检测、跟踪和描述。该任务统一了视频的空间与时间理解，并需要细粒度的语言描述。我们的密集视频对象描述模型采用端到端训练，包含空间定位、跟踪和描述等不同模块。因此，我们可以利用混合的非联合任务来训练模型，并借助标注不同模块的大规模多样化数据集。这带来了显著的零样本性能。此外，通过在该初始化基础上进行微调，我们可以进一步提升性能，大幅超越基于图像的强基线模型。尽管我们未发现其他相关工作，但成功将现有视频定位数据集（如VidSTG和VLN）重新用于此任务。我们证明，该任务比定位更具通用性，且基于该任务训练的模型可通过寻找生成查询语句的最大似然边界框直接应用于定位任务。在VidSTG和VLN数据集上，我们的模型在空间定位方面优于专用且最先进的模型。