Natural language explanations promise to offer intuitively understandable explanations of a neural network's decision process in complex vision-language tasks, as pursued in recent VL-NLE models. While current models offer impressive performance on task accuracy and explanation plausibility, they suffer from a range of issues: Some models feature a modular design where the explanation generation module is poorly integrated with a separate module for task-answer prediction, employ backbone models trained on limited sets of tasks, or incorporate ad hoc solutions to increase performance on single datasets. We propose to evade these limitations by applying recent advances in large-scale multi-task pretraining of generative Transformer models to the problem of VL-NLE tasks. Our approach outperforms recent models by a large margin, with human annotators preferring the generated explanations over the ground truth in two out of three evaluated datasets. As a novel challenge in VL-NLE research, we propose the problem of multi-task VL-NLE and show that jointly training on multiple tasks can increase the explanation quality. We discuss the ethical implications of high-quality NLE generation and other issues in recent VL-NLE research.

翻译：自然语言解释旨在为复杂的视觉-语言任务中神经网络的决策过程提供直观可理解的解释，这正是近期VL-NLE模型所追求的目标。尽管当前模型在任务准确性和解释合理性方面表现出色，但仍存在一系列问题：部分模型采用模块化设计，其中解释生成模块与独立的任务答案预测模块整合不佳；另一些模型依赖在有限任务集上训练的主干网络，或采用临时性方案以提升单一数据集上的性能。为规避这些局限性，我们提出将生成式Transformer模型在大规模多任务预训练方面的最新进展应用于VL-NLE任务。我们的方法在三个评估数据集中的两个上大幅超越近期模型，人工标注者甚至更倾向于选择生成的解释而非真实标签。作为VL-NLE研究中的新挑战，我们提出了多任务VL-NLE问题，并证明联合训练多个任务可提升解释质量。我们还讨论了高质量NLE生成的伦理影响及近期VL-NLE研究中的其他问题。