Medical caption prediction which can be regarded as a task of medical report generation (MRG), requires the automatic generation of coherent and accurate captions for the given medical images. However, the scarcity of labelled medical image-report pairs presents great challenges in the development of deep and large-scale neural networks capable of harnessing the potential artificial general intelligence power like large language models (LLMs). In this work, we propose customizing off-the-shelf general-purpose large-scale pre-trained models, i.e., foundation models (FMs), in computer vision and natural language processing with a specific focus on medical report generation. Specifically, following BLIP-2, a state-of-the-art vision-language pre-training approach, we introduce our encoder-decoder-based MRG model. This model utilizes a lightweight query Transformer to connect two FMs: the giant vision Transformer EVA-ViT-g and a bilingual LLM trained to align with human intentions (referred to as ChatGLM-6B). Furthermore, we conduct ablative experiments on the trainable components of the model to identify the crucial factors for effective transfer learning. Our findings demonstrate that unfreezing EVA-ViT-g to learn medical image representations, followed by parameter-efficient training of ChatGLM-6B to capture the writing styles of medical reports, is essential for achieving optimal results. Our best attempt (PCLmed Team) achieved the 4th and the 2nd, respectively, out of 13 participating teams, based on the BERTScore and ROUGE-1 metrics, in the ImageCLEFmedical Caption 2023 Caption Prediction Task competition.

翻译：医学标题预测可视为医学报告生成任务之一，需要针对给定的医学影像自动生成连贯且准确的描述。然而，标注的医学影像-报告对数据的稀缺性，对开发能够发挥大语言模型等潜在通用人工智能能力的深度大规模神经网络构成了重大挑战。本研究提出将计算机视觉与自然语言处理领域现成的通用大规模预训练模型（即基础模型）进行定制化改造，重点聚焦医学报告生成任务。具体而言，我们借鉴当前最先进的视觉-语言预训练方法BLIP-2，引入基于编码器-解码器的医学报告生成模型。该模型采用轻量级查询Transformer连接两大基础模型：巨型视觉Transformer EVA-ViT-g与经人类意图对齐训练的双语大语言模型（即ChatGLM-6B）。此外，我们通过消融实验探究模型可训练组件对迁移学习效果的关键影响因素。研究发现，解冻EVA-ViT-g以学习医学影像表征，再对ChatGLM-6B进行参数高效训练以掌握医学报告写作风格，是实现最优性能的必要条件。在ImageCLEFmedical Caption 2023标题预测任务竞赛中，我们的最佳方案（PCLmed团队）基于BERTScore和ROUGE-1指标，在13支参赛队伍中分别取得第4名与第2名的成绩。