Supervised learning aims to train a classifier under the assumption that training and test data are from the same distribution. To ease the above assumption, researchers have studied a more realistic setting: out-of-distribution (OOD) detection, where test data may come from classes that are unknown during training (i.e., OOD data). Due to the unavailability and diversity of OOD data, good generalization ability is crucial for effective OOD detection algorithms, and corresponding learning theory is still an open problem. To study the generalization of OOD detection, this paper investigates the probably approximately correct (PAC) learning theory of OOD detection that fits the commonly used evaluation metrics in the literature. First, we find a necessary condition for the learnability of OOD detection. Then, using this condition, we prove several impossibility theorems for the learnability of OOD detection under some scenarios. Although the impossibility theorems are frustrating, we find that some conditions of these impossibility theorems may not hold in some practical scenarios. Based on this observation, we next give several necessary and sufficient conditions to characterize the learnability of OOD detection in some practical scenarios. Lastly, we offer theoretical support for representative OOD detection works based on our OOD theory.

翻译：监督学习旨在假设训练数据和测试数据来自相同分布的情况下训练分类器。为了放宽上述假设，研究人员研究了更具现实意义的场景：分布外（OOD）检测，其中测试数据可能包含训练阶段未知的类别（即OOD数据）。由于OOD数据的不可获得性和多样性，良好的泛化能力对于有效的OOD检测算法至关重要，而相应的学习理论仍是一个开放问题。为了研究OOD检测的泛化性，本文探讨了与文献中常用评估指标相契合的分布外检测概率近似正确（PAC）学习理论。首先，我们发现了OOD检测可学习性的一个必要条件。然后，利用该条件，我们证明了在某些场景下OOD检测可学习性的若干不可能性定理。尽管这些不可能性定理令人沮丧，但我们发现这些定理的某些条件在实际场景中可能不成立。基于这一观察，我们接下来给出了若干充要条件，用于刻画某些实际场景中OOD检测的可学习性。最后，我们基于所提出的OOD理论，为具有代表性的OOD检测工作提供了理论支持。