The conventional pretraining-and-finetuning paradigm, while effective for common diseases with ample data, faces challenges in diagnosing data-scarce occupational diseases like pneumoconiosis. Recently, large language models (LLMs) have exhibits unprecedented ability when conducting multiple tasks in dialogue, bringing opportunities to diagnosis. A common strategy might involve using adapter layers for vision-language alignment and diagnosis in a dialogic manner. Yet, this approach often requires optimization of extensive learnable parameters in the text branch and the dialogue head, potentially diminishing the LLMs' efficacy, especially with limited training data. In our work, we innovate by eliminating the text branch and substituting the dialogue head with a classification head. This approach presents a more effective method for harnessing LLMs in diagnosis with fewer learnable parameters. Furthermore, to balance the retention of detailed image information with progression towards accurate diagnosis, we introduce the contextual multi-token engine. This engine is specialized in adaptively generating diagnostic tokens. Additionally, we propose the information emitter module, which unidirectionally emits information from image tokens to diagnosis tokens. Comprehensive experiments validate the superiority of our methods and the effectiveness of proposed modules. Our codes can be found at https://github.com/CodeMonsterPHD/PneumoLLM/tree/main.

翻译：传统的预训练与微调范式虽对数据充足的常见疾病行之有效，但在诊断数据稀缺的职业病（如尘肺病）时面临挑战。近期，大型语言模型在对话式多任务处理中展现出前所未有的能力，为诊断工作带来新机遇。常见策略可能涉及使用适配器模块实现视觉-语言对齐及对话式诊断。然而，这类方法通常需要在文本分支和对话头中优化大量可学习参数，可能削弱大语言模型的效能，尤其在训练数据有限的情况下。我们的工作创新性地移除文本分支，并用分类头替代对话头。该方案以更少的可学习参数实现了更高效的大语言模型诊断应用。此外，为平衡详细图像信息的保留与诊断准确性的提升，我们引入上下文多令牌引擎，该引擎专用于自适应生成诊断令牌。同时提出信息发射模块，实现从图像令牌到诊断令牌的单向信息传递。综合实验验证了本方法的优越性及所提模块的有效性。代码详见https://github.com/CodeMonsterPHD/PneumoLLM/tree/main。