The conventional pretraining-and-finetuning paradigm, while effective for common diseases with ample data, faces challenges in diagnosing data-scarce occupational diseases like pneumoconiosis. Recently, large language models (LLMs) have exhibits unprecedented ability when conducting multiple tasks in dialogue, bringing opportunities to diagnosis. A common strategy might involve using adapter layers for vision-language alignment and diagnosis in a dialogic manner. Yet, this approach often requires optimization of extensive learnable parameters in the text branch and the dialogue head, potentially diminishing the LLMs' efficacy, especially with limited training data. In our work, we innovate by eliminating the text branch and substituting the dialogue head with a classification head. This approach presents a more effective method for harnessing LLMs in diagnosis with fewer learnable parameters. Furthermore, to balance the retention of detailed image information with progression towards accurate diagnosis, we introduce the contextual multi-token engine. This engine is specialized in adaptively generating diagnostic tokens. Additionally, we propose the information emitter module, which unidirectionally emits information from image tokens to diagnosis tokens. Comprehensive experiments validate the superiority of our methods and the effectiveness of proposed modules. Our codes can be found at https://github.com/CodeMonsterPHD/PneumoLLM/tree/main.

翻译：摘要：传统的预训练-微调范式在应对数据充足的常见疾病时效果显著，但在诊断尘肺病等数据稀缺的职业病时面临挑战。近年来，大型语言模型在对话多任务处理中展现出前所未有的能力，为疾病诊断带来新机遇。一种常见策略是使用适配器层实现视觉语言对齐并以对话方式进行诊断，但该方法通常需要优化文本分支和对话头中的大量可学习参数，可能削弱大型语言模型的效能，尤其在训练数据有限的情况下。本研究通过移除文本分支并将对话头替换为分类头进行创新，提出了一种更高效利用大型语言模型进行诊断的方法，且所需可学习参数更少。此外，为平衡保留详细图像信息与推进精确诊断的需求，我们引入了上下文多令牌引擎，该引擎可自适应生成诊断令牌。同时，我们提出信息发射器模块，用于将图像令牌中的信息单向传输至诊断令牌。综合实验验证了所提方法的优越性及各模块的有效性。代码开源地址：https://github.com/CodeMonsterPHD/PneumoLLM/tree/main。