Video Question Answering (VideoQA) is a challenging task that entails complex multi-modal reasoning. In contrast to multiple-choice VideoQA which aims to predict the answer given several options, the goal of open-ended VideoQA is to answer questions without restricting candidate answers. However, the majority of previous VideoQA models formulate open-ended VideoQA as a classification task to classify the video-question pairs into a fixed answer set, i.e., closed-vocabulary, which contains only frequent answers (e.g., top-1000 answers). This leads the model to be biased toward only frequent answers and fail to generalize on out-of-vocabulary answers. We hence propose a new benchmark, Open-vocabulary Video Question Answering (OVQA), to measure the generalizability of VideoQA models by considering rare and unseen answers. In addition, in order to improve the model's generalization power, we introduce a novel GNN-based soft verbalizer that enhances the prediction on rare and unseen answers by aggregating the information from their similar words. For evaluation, we introduce new baselines by modifying the existing (closed-vocabulary) open-ended VideoQA models and improve their performances by further taking into account rare and unseen answers. Our ablation studies and qualitative analyses demonstrate that our GNN-based soft verbalizer further improves the model performance, especially on rare and unseen answers. We hope that our benchmark OVQA can serve as a guide for evaluating the generalizability of VideoQA models and inspire future research. Code is available at https://github.com/mlvlab/OVQA.

翻译：视频问答是一项具有挑战性的任务，涉及复杂的多模态推理。与旨在从多个选项中选择答案的多选式视频问答不同，开放域视频问答的目标是在不限制候选答案的情况下回答问题。然而，以往的大多数视频问答模型将开放域视频问答视作分类任务，将视频-问题对归类到固定的答案集合（即封闭词汇集）中，该集合仅包含高频答案（如前1000个答案）。这导致模型偏向于仅预测高频答案，而无法泛化到词汇表外的答案。为此，我们提出一个新基准——开放词汇视频问答，通过考虑罕见及未见过的答案来衡量视频问答模型的泛化能力。此外，为提升模型的泛化能力，我们引入一种基于图神经网络的软标签映射器，通过聚合相似词汇的信息来增强对罕见及未见答案的预测。在评估方面，我们通过修改现有的（封闭词汇）开放域视频问答模型建立了新基线，并进一步考虑了罕见和未见过的答案以提升其性能。我们的消融实验和定性分析表明，基于图神经网络的软标签映射器能进一步提升模型性能，尤其在处理罕见和未见答案时。我们希望我们的开放词汇视频问答基准能作为评估视频问答模型泛化能力的指南，并启发未来研究。代码已开源在 https://github.com/mlvlab/OVQA。