Transformers excel in natural language processing and computer vision tasks. However, they still face challenges in generalizing to Out-of-Distribution (OOD) datasets, i.e. data whose distribution differs from that seen during training. OOD detection aims to distinguish outliers while preserving in-distribution (ID) data performance. This paper introduces the OOD detection Probably Approximately Correct (PAC) Theory for transformers, which establishes the conditions for data distribution and model configurations for the OOD detection learnability of transformers. It shows that outliers can be accurately represented and distinguished with sufficient data under conditions. The theoretical implications highlight the trade-off between theoretical principles and practical training paradigms. By examining this trade-off, we naturally derived the rationale for leveraging auxiliary outliers to enhance OOD detection. Our theory suggests that by penalizing the misclassification of outliers within the loss function and strategically generating soft synthetic outliers, one can robustly bolster the reliability of transformer networks. This approach yields a novel algorithm that ensures learnability and refines the decision boundaries between inliers and outliers. In practice, the algorithm consistently achieves state-of-the-art (SOTA) performance across various data formats.

翻译：Transformer在自然语言处理和计算机视觉任务中表现出色。然而，其在泛化至分布外数据集时仍面临挑战，即处理与训练数据分布不同的数据。分布外检测旨在区分异常样本的同时保持分布内数据的性能。本文提出了针对Transformer的分布外检测近似正确理论，该理论建立了Transformer实现分布外检测可学习性所需的数据分布条件与模型配置条件。研究表明，在满足理论条件且数据充足的条件下，异常样本能够被准确表征与区分。理论分析揭示了理论原则与实际训练范式之间的权衡关系。通过考察这种权衡，我们自然推导出利用辅助异常样本来增强分布外检测的理论依据。该理论表明，通过在损失函数中对异常样本误分类施加惩罚，并策略性地生成软合成异常样本，能够有效增强Transformer网络的可靠性。该方法催生了一种保证可学习性并优化正常样本与异常样本间决策边界的新算法。实验表明，该算法在多种数据模态上均能持续取得最先进的性能表现。