In recent years, artificial intelligence has played an important role in medicine and disease diagnosis, with many applications to be mentioned, one of which is Medical Visual Question Answering (MedVQA). By combining computer vision and natural language processing, MedVQA systems can assist experts in extracting relevant information from medical image based on a given question and providing precise diagnostic answers. The ImageCLEFmed-MEDVQA-GI-2023 challenge carried out visual question answering task in the gastrointestinal domain, which includes gastroscopy and colonoscopy images. Our team approached Task 1 of the challenge by proposing a multimodal learning method with image enhancement to improve the VQA performance on gastrointestinal images. The multimodal architecture is set up with BERT encoder and different pre-trained vision models based on convolutional neural network (CNN) and Transformer architecture for features extraction from question and endoscopy image. The result of this study highlights the dominance of Transformer-based vision models over the CNNs and demonstrates the effectiveness of the image enhancement process, with six out of the eight vision models achieving better F1-Score. Our best method, which takes advantages of BERT+BEiT fusion and image enhancement, achieves up to 87.25% accuracy and 91.85% F1-Score on the development test set, while also producing good result on the private test set with accuracy of 82.01%.

翻译：近年来，人工智能在医学与疾病诊断领域发挥了重要作用，诸多应用场景中值得关注的一项便是医学视觉问答（MedVQA）。通过结合计算机视觉与自然语言处理技术，MedVQA系统能够基于给定问题辅助专家从医学图像中提取相关信息，并提供精确的诊断答案。ImageCLEFmed-MEDVQA-GI-2023挑战赛聚焦于胃肠道领域的视觉问答任务，数据集涵盖胃镜与结肠镜图像。本团队通过提出一种结合图像增强的多模态学习方法，以提升胃肠道图像的VQA性能。该多模态架构采用BERT编码器，并基于卷积神经网络（CNN）与Transformer架构的不同预训练视觉模型，分别提取问题与内镜图像的特征。研究结果凸显了基于Transformer的视觉模型相较CNN的显著优势，同时验证了图像增强处理的有效性——在八种视觉模型中，有六种通过该方法获得了更高的F1分数。我们表现最佳的方法结合了BERT+BEiT融合与图像增强技术，在开发测试集上实现了高达87.25%的准确率与91.85%的F1分数，并在私有测试集上以82.01%的准确率取得了优异表现。