The main challenge in video question answering (VideoQA) is to capture and understand the complex spatial and temporal relations between objects based on given questions. Existing graph-based methods for VideoQA usually ignore keywords in questions and employ a simple graph to aggregate features without considering relative relations between objects, which may lead to inferior performance. In this paper, we propose a Keyword-aware Relative Spatio-Temporal (KRST) graph network for VideoQA. First, to make question features aware of keywords, we employ an attention mechanism to assign high weights to keywords during question encoding. The keyword-aware question features are then used to guide video graph construction. Second, because relations are relative, we integrate the relative relation modeling to better capture the spatio-temporal dynamics among object nodes. Moreover, we disentangle the spatio-temporal reasoning into an object-level spatial graph and a frame-level temporal graph, which reduces the impact of spatial and temporal relation reasoning on each other. Extensive experiments on the TGIF-QA, MSVD-QA and MSRVTT-QA datasets demonstrate the superiority of our KRST over multiple state-of-the-art methods.

翻译：视频问答（VideoQA）的主要挑战在于基于给定问题捕获并理解对象间复杂的时空关系。现有基于图的VideoQA方法通常忽略问题中的关键词，并采用简单图聚合特征而未考虑对象间的相对关系，这可能导致性能欠佳。本文提出一种关键词感知的相对时空（KRST）图网络用于VideoQA。首先，为使问题特征感知关键词，我们在问题编码过程中采用注意力机制为关键词赋予高权重。随后，关键词感知的问题特征被用于指导视频图构建。其次，由于关系具有相对性，我们整合相对关系建模以更好捕获对象节点间的时空动态。此外，我们将时空推理解耦为对象级空间图和帧级时间图，从而降低空间与时间关系推理间的相互影响。在TGIF-QA、MSVD-QA和MSRVTT-QA数据集上的大量实验表明，我们的KRST方法在多个最先进方法中展现出优越性。