Out-of-distribution (OOD) detection is critical for safety-sensitive machine learning applications and has been extensively studied, yielding a plethora of methods developed in the literature. However, most studies for OOD detection did not use pre-trained models and trained a backbone from scratch. In recent years, transferring knowledge from large pre-trained models to downstream tasks by lightweight tuning has become mainstream for training in-distribution (ID) classifiers. To bridge the gap between the practice of OOD detection and current classifiers, the unique and crucial problem is that the samples whose information networks know often come as OOD input. We consider that such data may significantly affect the performance of large pre-trained networks because the discriminability of these OOD data depends on the pre-training algorithm. Here, we define such OOD data as PT-OOD (Pre-Trained OOD) data. In this paper, we aim to reveal the effect of PT-OOD on the OOD detection performance of pre-trained networks from the perspective of pre-training algorithms. To achieve this, we explore the PT-OOD detection performance of supervised and self-supervised pre-training algorithms with linear-probing tuning, the most common efficient tuning method. Through our experiments and analysis, we find that the low linear separability of PT-OOD in the feature space heavily degrades the PT-OOD detection performance, and self-supervised models are more vulnerable to PT-OOD than supervised pre-trained models, even with state-of-the-art detection methods. To solve this vulnerability, we further propose a unique solution to large-scale pre-trained models: Leveraging powerful instance-by-instance discriminative representations of pre-trained models and detecting OOD in the feature space independent of the ID decision boundaries. The code will be available via https://github.com/AtsuMiyai/PT-OOD.

翻译：分布外（OOD）检测对于安全敏感的机器学习应用至关重要，并已得到广泛研究，文献中涌现了大量方法。然而，大多数OOD检测研究并未使用预训练模型，而是从头训练骨干网络。近年来，通过轻量级微调将大规模预训练模型的知识迁移到下游任务，已成为训练分布内（ID）分类器的主流方法。为弥合OOD检测实践与当前分类器之间的差距，一个独特且关键的问题在于：网络已知信息对应的样本通常以分布外输入的形式出现。我们认为此类数据可能显著影响大规模预训练网络的性能，因为这类OOD数据的可区分性取决于预训练算法。这里将此类OOD数据定义为PT-OOD（预训练分布外）数据。本文旨在从预训练算法的角度揭示PT-OOD对预训练网络OOD检测性能的影响。为此，我们探究了采用线性探测微调（最常用的高效微调方法）的有监督和自监督预训练算法的PT-OOD检测性能。通过实验与分析发现，PT-OOD在特征空间中的低线性可分性严重削弱了其检测性能，且自监督模型比有监督预训练模型对PT-OOD更敏感，即便采用最先进的检测方法也是如此。针对这一脆弱性，我们进一步提出一种适用于大规模预训练模型的独特解决方案：利用预训练模型强大的逐实例判别表示，在独立于ID决策边界的特征空间中检测OOD。代码将通过 https://github.com/AtsuMiyai/PT-OOD 提供。