Visual Question Answering (VQA) is a challenging task that requires cross-modal understanding and reasoning of visual image and natural language question. To inspect the association of VQA models to human cognition, we designed a survey to record human thinking process and analyzed VQA models by comparing the outputs and attention maps with those of humans. We found that although the VQA models resemble human cognition in architecture and performs similarly with human on the recognition-level, they still struggle with cognitive inferences. The analysis of human thinking procedure serves to direct future research and introduce more cognitive capacity into modeling features and architectures.

翻译：视觉问答（Visual Question Answering, VQA）是一项需要跨模态理解与推理（涉及视觉图像和自然语言问题）的挑战性任务。为了探究VQA模型与人类认知的关联，我们设计了一项调查来记录人类的思维过程，并通过比较模型输出及注意力图与人类对应结果的方式对VQA模型进行分析。研究发现，尽管VQA模型在架构上近似人类认知，且在识别层面表现出与人类相似的能力，但在认知推理方面仍存在明显不足。对人类思维过程的分析有助于指导未来研究，并推动建模特征与架构中融入更多认知能力。