Detecting Out-of-distribution (OOD) inputs have been a critical issue for neural networks in the open world. However, the unstable behavior of OOD detection along the optimization trajectory during training has not been explored clearly. In this paper, we first find the performance of OOD detection suffers from overfitting and instability during training: 1) the performance could decrease when the training error is near zero, and 2) the performance would vary sharply in the final stage of training. Based on our findings, we propose Average of Pruning (AoP), consisting of model averaging and pruning, to mitigate the unstable behaviors. Specifically, model averaging can help achieve a stable performance by smoothing the landscape, and pruning is certified to eliminate the overfitting by eliminating redundant features. Comprehensive experiments on various datasets and architectures are conducted to verify the effectiveness of our method.

翻译：检测开放世界中神经网络的分布外输入一直是关键问题，然而训练过程中优化轨迹对分布外检测的不稳定行为尚未得到清晰探究。本文首先发现，分布外检测性能在训练中会出现过拟合与不稳定性：1) 当训练误差趋近于零时，检测性能反而下降；2) 训练末期检测性能波动剧烈。基于此发现，我们提出包含模型平均与剪枝的"平均剪枝"方法以缓解不稳定行为。具体而言，模型平均通过平滑损失景观实现稳定性能，剪枝则被验证可通过消除冗余特征来抑制过拟合。我们在多种数据集与架构上进行了全面实验，验证了所提方法的有效性。