Out-of-Distribution (OOD) detection is a critical task that has garnered significant attention. The emergence of CLIP has spurred extensive research into zero-shot OOD detection, often employing a training-free approach. Current methods leverage expert knowledge from large language models (LLMs) to identify potential outliers. However, these approaches tend to over-rely on knowledge in the text space, neglecting the inherent challenges involved in detecting out-of-distribution samples in the image space. In this paper, we propose a novel pipeline, MM-OOD, which leverages the multimodal reasoning capabilities of MLLMs and their ability to conduct multi-round conversations for enhanced outlier detection. Our method is designed to improve performance in both near OOD and far OOD tasks. Specifically, (1) for near OOD tasks, we directly feed ID images and corresponding text prompts into MLLMs to identify potential outliers; and (2) for far OOD tasks, we introduce the sketch-generate-elaborate framework: first, we sketch outlier exposure using text prompts, then generate corresponding visual OOD samples, and finally elaborate by using multimodal prompts. Experiments demonstrate that our method achieves significant improvements on widely used multimodal datasets such as Food-101, while also validating its scalability on ImageNet-1K.

翻译：分布外（OOD）检测是一项受到广泛关注的关键任务。CLIP的出现推动了零样本OOD检测的深入研究，通常采用免训练方法。现有方法利用大语言模型（LLMs）的专家知识来识别潜在异常样本。然而，这些方法往往过度依赖文本空间的知识，忽视了图像空间中检测分布外样本所固有的挑战。本文提出一种新颖的流程MM-OOD，它利用多模态大语言模型（MLLMs）的多模态推理能力及其进行多轮对话的功能，以增强异常检测。我们的方法旨在提升近分布外与远分布外任务的性能。具体而言：（1）对于近分布外任务，我们直接将分布内图像及相应文本提示输入MLLMs以识别潜在异常样本；（2）对于远分布外任务，我们引入草图-生成-精修框架：首先通过文本提示勾勒异常暴露，随后生成相应的视觉OOD样本，最后利用多模态提示进行精修。实验表明，我们的方法在Food-101等广泛使用的多模态数据集上取得了显著改进，同时也在ImageNet-1K上验证了其可扩展性。