A well-known failure mode of neural networks is that they may confidently return erroneous predictions. Such unsafe behaviour is particularly frequent when the use case slightly differs from the training context, and/or in the presence of an adversary. This work presents a novel direction to address these issues in a broad, general manner: imposing class-aware constraints on a model's internal activation patterns. Specifically, we assign to each class a unique, fixed, randomly-generated binary vector - hereafter called class code - and train the model so that its cross-depths activation patterns predict the appropriate class code according to the input sample's class. The resulting predictors are dubbed Total Activation Classifiers (TAC), and TACs may either be trained from scratch, or used with negligible cost as a thin add-on on top of a frozen, pre-trained neural network. The distance between a TAC's activation pattern and the closest valid code acts as an additional confidence score, besides the default unTAC'ed prediction head's. In the add-on case, the original neural network's inference head is completely unaffected (so its accuracy remains the same) but we now have the option to use TAC's own confidence and prediction when determining which course of action to take in an hypothetical production workflow. In particular, we show that TAC strictly improves the value derived from models allowed to reject/defer. We provide further empirical evidence that TAC works well on multiple types of architectures and data modalities and that it is at least as good as state-of-the-art alternative confidence scores derived from existing models.

翻译：神经网络的一个众所周知的失效模式是它们可能自信地返回错误预测。这种不安全行为在使用场景与训练环境稍有不同时，或存在对抗性攻击的情况下尤为常见。本文提出了一种广泛通用的新方向来解决这些问题：对模型内部激活模式施加类别感知约束。具体而言，我们为每个类别分配一个唯一的、固定的、随机生成的二进制向量（以下称为类别编码），并训练模型，使其跨层激活模式根据输入样本的类别预测相应的类别编码。由此产生的预测器被称为总激活分类器（TAC），TAC既可以从头训练，也可以作为轻量附加模块以极低成本叠加在冻结的预训练神经网络上。TAC激活模式与最近有效编码之间的距离可作为一种额外的置信度评分，与默认的未TAC预测头输出的置信度互补。在附加模块场景中，原始神经网络的推理头完全不受影响（因此其准确率保持不变），但在假设的生产工作流中，我们可以选择使用TAC自身的置信度和预测来决定采取何种行动。特别地，我们证明TAC能够严格提升允许拒绝/延迟决策的模型的价值。我们进一步提供了实证证据，表明TAC在多种架构和数据模态下均表现良好，且其性能至少与现有模型导出的最先进替代置信度评分相当。