Large Language Models (LLMs) have introduced a new era of proficiency in comprehending complex healthcare and biomedical topics. However, there is a noticeable lack of models in languages other than English and models that can interpret multi-modal input, which is crucial for global healthcare accessibility. In response, this study introduces Qilin-Med-VL, the first Chinese large vision-language model designed to integrate the analysis of textual and visual data. Qilin-Med-VL combines a pre-trained Vision Transformer (ViT) with a foundational LLM. It undergoes a thorough two-stage curriculum training process that includes feature alignment and instruction tuning. This method enhances the model's ability to generate medical captions and answer complex medical queries. We also release ChiMed-VL, a dataset consisting of more than 1M image-text pairs. This dataset has been carefully curated to enable detailed and comprehensive interpretation of medical data using various types of images.

翻译：大型语言模型（LLMs）在理解复杂医疗和生物医学主题方面开启了全新的能力时代。然而，非英语语言模型及能解读多模态输入（对全球医疗可及性至关重要）的模型明显匮乏。为此，本研究提出Qilin-Med-VL——首个专为整合文本与视觉数据分析而设计的中文大型视觉语言模型。该模型结合了预训练的视觉Transformer（ViT）与基础LLM，并通过包含特征对齐与指令微调的完整两阶段课程训练流程进行优化。此方法增强了模型生成医学描述与解答复杂医学查询的能力。我们还发布了包含超过100万图像-文本对的ChiMed-VL数据集。该数据集经过精心筛选，旨在利用多种类型图像实现对医疗数据的详尽与全面解读。