In the realm of Medical Visual Language Models (Med-VLMs), the quest for universal efficient fine-tuning mechanisms remains paramount, especially given researchers in interdisciplinary fields are often extremely short of training resources, yet largely unexplored. Given the unique challenges in the medical domain, such as limited data scope and significant domain-specific requirements, evaluating and adapting Parameter-Efficient Fine-Tuning (PEFT) methods specifically for Med-VLMs is essential. Most of the current PEFT methods on Med-VLMs have yet to be comprehensively investigated but mainly focus on adding some components to the model's structure or input. However, fine-tuning intrinsic model components often yields better generality and consistency, and its impact on the ultimate performance of Med-VLMs has been widely overlooked and remains understudied. In this paper, we endeavour to explore an alternative to traditional PEFT methods, especially the impact of fine-tuning LayerNorm layers, FFNs and Attention layers on the Med-VLMs. Our comprehensive studies span both small-scale and large-scale Med-VLMs, evaluating their performance under various fine-tuning paradigms across tasks such as Medical Visual Question Answering and Medical Imaging Report Generation. The findings reveal unique insights into the effects of intrinsic parameter fine-tuning methods on fine-tuning Med-VLMs to downstream tasks and expose fine-tuning solely the LayerNorm layers not only surpasses the efficiency of traditional PEFT methods but also retains the model's accuracy and generalization capabilities across a spectrum of medical downstream tasks. The experiments show LayerNorm fine-tuning's superior adaptability and scalability, particularly in the context of large-scale Med-VLMs.

翻译：在医学视觉语言模型（Med-VLMs）领域，寻找通用的高效微调机制仍然至关重要，尤其是考虑到跨学科领域的研究人员通常极度缺乏训练资源，但这一方向很大程度上未被探索。鉴于医学领域的独特挑战，例如有限的数据范围和显著的领域特异性需求，评估和适配专门针对Med-VLMs的参数高效微调（PEFT）方法至关重要。目前针对Med-VLMs的大多数PEFT方法尚未得到全面研究，主要集中于在模型结构或输入中添加某些组件。然而，微调模型内在组件往往能带来更好的通用性和一致性，但其对Med-VLM最终性能的影响却被广泛忽视且研究不足。在本文中，我们致力于探索传统PEFT方法的替代方案，特别是微调层归一化（LayerNorm）层、前馈神经网络（FFN）和注意力（Attention）层对Med-VLM的影响。我们的综合研究涵盖了从小规模到大规模的Med-VLMs，评估了它们在各种微调范式下于医学视觉问答和医学影像报告生成等任务中的表现。研究结果揭示了内在参数微调方法对Med-VLM下游任务适配的独特见解，并发现仅微调LayerNorm层不仅超越了传统PEFT方法的效率，还能在多种医学下游任务中保持模型的准确性和泛化能力。实验表明，LayerNorm微调具有卓越的适应性和可扩展性，尤其是在大规模Med-VLM的背景下。