Multimodal large language models (MLLMs) represent an evolutionary expansion in the capabilities of traditional large language models, enabling them to tackle challenges that surpass the scope of purely text-based applications. It leverages the knowledge previously encoded within these language models, thereby enhancing their applicability and functionality in the reign of multimodal contexts. Recent works investigate the adaptation of MLLMs to predict free-form answers as a generative task to solve medical visual question answering (Med-VQA) tasks. In this paper, we propose a parameter efficient framework for fine-tuning MLLM specifically tailored to Med-VQA applications, and empirically validate it on a public benchmark dataset. To accurately measure the performance, we employ human evaluation and the results reveal that our model achieves an overall accuracy of 81.9%, and outperforms the GPT-4v model by a significant margin of 26% absolute accuracy on closed-ended questions. The code will be available here: https://github.com/jinlHe/PeFoMed.

翻译：多模态大语言模型（MLLMs）代表了传统大语言模型能力的进化扩展，使其能够应对超越纯文本应用范畴的挑战。这类模型充分利用语言模型中预编码的知识，从而在多模态语境中增强了适用性与功能性。近期研究探索将MLLMs适配为生成式任务，通过预测自由形式答案来求解医学视觉问答（Med-VQA）任务。本文针对Med-VQA应用场景，提出了一种面向MLLMs微调的参数高效框架，并在公开基准数据集上进行了实证验证。为精准评估性能，我们采用人工评估方法，结果显示：所提模型整体准确率达81.9%，且在封闭式问题上以26%的绝对准确率显著超越GPT-4v模型。代码将开源于：https://github.com/jinlHe/PeFoMed。