An important aspect of summarizing videos is understanding the temporal context behind each part of the video to grasp what is and is not important. Video summarization models have in recent years modeled spatio-temporal relationships to represent this information. These models achieved state-of-the-art correlation scores on important benchmark datasets. However, what has not been reviewed is whether spatio-temporal relationships are even required to achieve state-of-the-art results. Previous work in activity recognition has found biases, by prioritizing static cues such as scenes or objects, over motion information. In this paper we inquire if similar spurious relationships might influence the task of video summarization. To do so, we analyse the role that temporal information plays on existing benchmark datasets. We first estimate a baseline with temporally invariant models to see how well such models rank on benchmark datasets (TVSum and SumMe). We then disrupt the temporal order of the videos to investigate the impact it has on existing state-of-the-art models. One of our findings is that the temporally invariant models achieve competitive correlation scores that are close to the human baselines on the TVSum dataset. We also demonstrate that existing models are not affected by temporal perturbations. Furthermore, with certain disruption strategies that shuffle fixed time segments, we can actually improve their correlation scores. With these results, we find that spatio-temporal relationship play a minor role and we raise the question whether these benchmarks adequately model the task of video summarization. Code available at: https://github.com/AashGan/TemporalPerturbSum

翻译：视频摘要的一个重要方面是理解视频各部分背后的时序上下文，以把握内容的重要性。近年来，视频摘要模型通过建模时空关系来表示此类信息，这些模型在重要基准数据集上取得了最先进的相关性分数。然而，尚未被审视的是：实现最先进结果是否真的需要时空关系。先前在行为识别领域的研究发现，模型存在优先考虑场景或物体等静态线索而忽视运动信息的偏差。本文旨在探究类似的伪相关关系是否可能影响视频摘要任务。为此，我们分析了现有基准数据集中时序信息的作用。首先，我们使用时序不变模型建立基线，评估此类模型在基准数据集（TVSum 和 SumMe）上的排名表现。随后，我们通过打乱视频的时序顺序来研究其对现有最先进模型的影响。我们的发现之一是：在 TVSum 数据集上，时序不变模型取得了接近人类基线的竞争性相关性分数。我们还证明现有模型不受时序扰动的影响。此外，通过采用打乱固定时间片段的特定干扰策略，我们甚至能提升模型的相关性分数。基于这些结果，我们认为时空关系仅起到次要作用，并由此提出疑问：这些基准测试是否充分建模了视频摘要任务。代码发布于：https://github.com/AashGan/TemporalPerturbSum