This paper explores post-disaster analytics using multimodal deep learning models trained with curriculum learning method. Studying post-disaster analytics is important as it plays a crucial role in mitigating the impact of disasters by providing timely and accurate insights into the extent of damage and the allocation of resources. We propose a curriculum learning strategy to enhance the performance of multimodal deep learning models. Curriculum learning emulates the progressive learning sequence in human education by training deep learning models on increasingly complex data. Our primary objective is to develop a curriculum-trained multimodal deep learning model, with a particular focus on visual question answering (VQA) capable of jointly processing image and text data, in conjunction with semantic segmentation for disaster analytics using the FloodNet\footnote{https://github.com/BinaLab/FloodNet-Challenge-EARTHVISION2021} dataset. To achieve this, U-Net model is used for semantic segmentation and image encoding. A custom built text classifier is used for visual question answering. Existing curriculum learning methods rely on manually defined difficulty functions. We introduce a novel curriculum learning approach termed Dynamic Task and Weight Prioritization (DATWEP), which leverages a gradient-based method to automatically decide task difficulty during curriculum learning training, thereby eliminating the need for explicit difficulty computation. The integration of DATWEP into our multimodal model shows improvement on VQA performance. Source code is available at https://github.com/fualsan/DATWEP.

翻译：本文探索了基于课程学习方法训练多模态深度学习模型的后灾害分析应用。研究后灾害分析至关重要，因其通过提供关于灾害破坏程度与资源分配的及时准确洞察，在减轻灾害影响方面发挥关键作用。我们提出了一种课程学习策略以提升多模态深度学习模型的性能。课程学习通过让深度学习模型在复杂度递增的数据上进行训练，模拟人类教育中的渐进式学习序列。我们的主要目标是开发一个经过课程训练的多模态深度学习模型，重点关注能够联合处理图像和文本数据的视觉问答（VQA），并结合语义分割技术，利用FloodNet数据集（来源：https://github.com/BinaLab/FloodNet-Challenge-EARTHVISION2021）进行灾害分析。为此，我们使用U-Net模型进行语义分割和图像编码，并采用自定义文本分类器进行视觉问答。现有课程学习方法依赖人工定义的难度函数。我们提出一种名为动态任务与权重优先级（DATWEP）的新型课程学习方法，该方法利用基于梯度的方法自动决定课程学习训练过程中的任务难度，从而消除显式难度计算的需求。将DATWEP集成到多模态模型中后，VQA性能得到了提升。源代码可从 https://github.com/fualsan/DATWEP 获取。