Recently we have witnessed the rapid development of video question answering models. However, most models can only handle simple videos in terms of temporal reasoning, and their performance tends to drop when answering temporal-reasoning questions on long and informative videos. To tackle this problem we propose STAIR, a Spatial-Temporal Reasoning model with Auditable Intermediate Results for video question answering. STAIR is a neural module network, which contains a program generator to decompose a given question into a hierarchical combination of several sub-tasks, and a set of lightweight neural modules to complete each of these sub-tasks. Though neural module networks are already widely studied on image-text tasks, applying them to videos is a non-trivial task, as reasoning on videos requires different abilities. In this paper, we define a set of basic video-text sub-tasks for video question answering and design a set of lightweight modules to complete them. Different from most prior works, modules of STAIR return intermediate outputs specific to their intentions instead of always returning attention maps, which makes it easier to interpret and collaborate with pre-trained models. We also introduce intermediate supervision to make these intermediate outputs more accurate. We conduct extensive experiments on several video question answering datasets under various settings to show STAIR's performance, explainability, compatibility with pre-trained models, and applicability when program annotations are not available. Code: https://github.com/yellow-binary-tree/STAIR

翻译：近期，视频问答模型取得了快速发展。然而，大多数模型在处理时间推理任务时仅能应对简单视频，在回答长时程、高信息量视频中的时间推理问题时性能往往下降。针对这一问题，我们提出STAIR——一种带有可审计中间结果的时空推理模型。STAIR是一种神经模块网络，包含一个程序生成器，用于将给定问题分解为若干子任务的分层组合，以及一组轻量级神经模块来完成每个子任务。尽管神经模块网络已在图文任务中得到广泛研究，但将其应用于视频领域并非易事，因为视频推理需要不同的能力。本文为视频问答定义了一组基础视频-文本子任务，并设计了相应的轻量级模块。与大多数先前工作不同，STAIR的模块始终返回与其意图对应的中间输出而非注意力图，这使得模型更易解释且便于与预训练模型协作。我们还引入中间监督机制以提升中间输出的准确性。在多种设置下的多个视频问答数据集上进行的广泛实验表明，STAIR在性能、可解释性、与预训练模型的兼容性以及缺乏程序标注时的适用性方面均表现优异。代码：https://github.com/yellow-binary-tree/STAIR