Although visual foundation models like DINOv2 provide state-of-the-art performance as feature extractors, their complex, high-dimensional representations create substantial hurdles for interpretability. This work proposes DINO-QPM, which converts these powerful but entangled features into contrastive, class-independent representations that are interpretable by humans. DINO-QPM is a lightweight interpretability adapter that pursues globally interpretable image classification, adapting the Quadratic Programming Enhanced Model (QPM) to operate on strictly frozen DINO backbones. While classification with visual foundation models typically relies on the \texttt{CLS} token, we deliberately diverge from this standard. By leveraging average-pooling, we directly connect the patch embeddings to the model's features and therefore enable spatial localisation of DINO-QPM's globally interpretable features within the input space. Furthermore, we apply a sparsity loss to minimise spatial scatter and background noise, ensuring that explanations are grounded in relevant object parts. With DINO-QPM we make the level of interpretability of QPM available as an adapter while exceeding the accuracy of DINOv2 linear probe. Evaluated through an introduced Plausibility metric and other interpretability metrics, extensive experiments demonstrate that DINO-QPM is superior to other applicable methods for frozen visual foundation models in both classification accuracy and explanation quality.
翻译:尽管像DINOv2这样的视觉基础模型作为特征提取器提供了最先进的性能,但其复杂的高维表示为可解释性带来了重大挑战。本文提出DINO-QPM,该方法将强大但纠缠的特征转换为具有对比性、类无关且人类可解释的表征。DINO-QPM是一种轻量级可解释性适配器,通过将二次规划增强模型(QPM)适配到完全冻结的DINO骨干网络上,实现全局可解释的图像分类。虽然视觉基础模型的分类通常依赖\texttt{CLS}令牌,我们有意背离这一标准做法。通过利用平均池化,我们将补丁嵌入直接连接到模型特征,从而能够在输入空间中定位DINO-QPM全局可解释特征的空间位置。此外,我们应用稀疏性损失以最小化空间分散和背景噪声,确保解释基于相关目标部件。借助DINO-QPM,我们使QPM的可解释性水平以适配器形式可用,同时超越DINOv2线性探测的准确率。通过引入的合理性指标及其他可解释性指标进行评估,大量实验表明,DINO-QPM在分类准确率和解释质量方面均优于其他适用于冻结视觉基础模型的方法。