Recent vision language models (VLMs) like CLIP have demonstrated impressive anomaly detection performance under significant distribution shift by utilizing high-level semantic information through text prompts. However, these models often neglect fine-grained details, such as which kind of anomalies, like "hole", "cut", "scratch" that could provide more specific insight into the nature of anomalies. We argue that recognizing fine-grained anomaly types 1) enriches the representation of "abnormal" with structured semantics, narrowing the gap between coarse anomaly signals and fine-grained defect categories; 2) enables manufacturers to understand the root causes of the anomaly and implement more targeted and appropriate corrective measures quickly. While incorporating such detailed semantic information is crucial, designing handcrafted prompts for each defect type is both time-consuming and susceptible to human bias. For this reason, we introduce DAPO, a novel approach for Defect-aware Prompt Optimization based on progressive tuning for the zero-shot multi-type and binary anomaly detection and segmentation under distribution shifts. Our approach aligns anomaly-relevant image features with their corresponding text semantics by learning hybrid defect-aware prompts with both fixed textual anchors and learnable token embeddings. We conducted experiments on public benchmarks (MPDD, VisA, MVTec-AD, MAD, and Real-IAD) and an internal dataset. The results suggest that compared to the baseline models, DAPO achieves a 3.7% average improvement in AUROC and average precision metrics at the image level under distribution shift, and a 6.5% average improvement in localizing novel anomaly types under zero-shot settings.

翻译：近期视觉语言模型（如CLIP）通过文本提示利用高层语义信息，在显著分布偏移下展现出优异的异常检测性能。然而，这些模型往往忽略细粒度细节（如“孔洞”“切口”“划痕”等具体异常类型），而这些信息可为异常本质提供更具体的洞察。我们认为识别细粒度异常类型具有双重意义：1）通过结构化语义丰富“异常”的表征，缩小粗粒度异常信号与细粒度缺陷类别之间的差距；2）帮助制造商理解异常根源，快速实施更具针对性且恰当的纠正措施。尽管融入此类细节语义信息至关重要，但为每种缺陷类型手工设计提示既耗时又易受人为偏差影响。为此，我们提出DAPO——一种基于渐进调优的缺陷感知提示优化新方法，用于分布偏移下的零样本多类型及二元异常检测与分割。该方法通过同时学习固定文本锚点与可学习词元嵌入构成的混合缺陷感知提示，将异常相关图像特征与其对应文本语义对齐。我们在公开基准数据集（MPDD、VisA、MVTec-AD、MAD、Real-IAD）及内部数据集上进行实验。结果表明：相较于基线模型，DAPO在分布偏移下的图像级AUROC与平均精度指标平均提升3.7%，在零样本设置下定位新型异常类型的性能平均提升6.5%。