Recently, there has been a growing interest in Multimodal Large Language Models (MLLMs) due to their remarkable potential in various tasks integrating different modalities, such as image and text, as well as applications such as image captioning and visual question answering. However, such models still face challenges in accurately captioning and interpreting specific visual concepts and classes, particularly in domain-specific applications. We argue that integrating domain knowledge in the form of an ontology can significantly address these issues. In this work, as a proof of concept, we propose a new framework that combines ontology with MLLMs to classify images of plant diseases. Our method uses concepts about plant diseases from an existing disease ontology to query MLLMs and extract relevant visual concepts from images. Then, we use the reasoning capabilities of the ontology to classify the disease according to the identified concepts. Ensuring that the model accurately uses the concepts describing the disease is crucial in domain-specific applications. By employing an ontology, we can assist in verifying this alignment. Additionally, using the ontology's inference capabilities increases transparency, explainability, and trust in the decision-making process while serving as a judge by checking if the annotations of the concepts by MLLMs are aligned with those in the ontology and displaying the rationales behind their errors. Our framework offers a new direction for synergizing ontologies and MLLMs, supported by an empirical study using different well-known MLLMs.

翻译：近年来，多模态大语言模型因其在图像与文本等多模态融合任务以及图像描述生成、视觉问答等应用中的卓越潜力而受到广泛关注。然而，此类模型在准确描述和解释特定视觉概念与类别方面仍面临挑战，尤其在领域专用应用中。我们认为，以本体形式整合领域知识可显著解决这些问题。本研究作为概念验证，提出一种将本体与多模态大语言模型相结合的新框架，用于植物病害图像分类。该方法利用现有病害本体中的植物病害概念查询多模态大语言模型，并从图像中提取相关视觉概念。随后，借助本体的推理能力，根据识别出的概念对病害进行分类。在领域专用应用中，确保模型准确使用描述病害的概念至关重要。通过采用本体，我们能够协助验证这种一致性。此外，利用本体的推理能力可增强决策过程的透明度、可解释性与可信度，同时通过检查多模态大语言模型的概念标注是否与本体对齐并展示其错误背后的逻辑依据，发挥评判作用。我们的框架为协同整合本体与多模态大语言模型提供了新方向，并通过采用不同知名多模态大语言模型的实证研究予以支持。