Having revolutionized natural language processing (NLP) applications, large language models (LLMs) are expanding into the realm of multimodal inputs. Owing to their ability to interpret images, multimodal LLMs (MLLMs) have been primarily used for vision-language tasks. Currently, MLLMs have not yet been extended for domain-specific visual tasks, which require a more explicit understanding of visual information. We developed a method to transform domain-specific visual and vision-language datasets into a unified question answering format called Visual Question Answering Instruction (VQA-IN), thereby extending MLLM to domain-specific tasks. The VQA-IN was applied to train multiple MLLM architectures using smaller versions of LLMs (sLLMs). The experimental results indicated that the proposed method achieved a high score metric on domainspecific visual tasks while also maintaining its performance on vision-language tasks in a multitask manner.

翻译：在彻底革新自然语言处理应用后，大语言模型正扩展至多模态输入领域。凭借图像解读能力，多模态大语言模型已主要应用于视觉-语言任务。目前，多模态大语言模型尚未扩展至需要更明确视觉信息理解的领域特定视觉任务。我们提出一种方法，将领域特定的视觉和视觉-语言数据集转化为统一的问答格式——视觉问答指令（VQA-IN），从而将多模态大语言模型扩展至领域特定任务。采用较小版本的大语言模型（sLLM），基于VQA-IN训练了多种多模态大语言模型架构。实验结果表明，该方法在领域特定视觉任务上取得了高评分指标，同时以多任务方式保持了在视觉-语言任务上的性能。