A Bayesian treatment can mitigate overconfidence in ReLU nets around the training data. But far away from them, ReLU Bayesian neural networks (BNNs) can still underestimate uncertainty and thus be asymptotically overconfident. This issue arises since the output variance of a BNN with finitely many features is quadratic in the distance from the data region. Meanwhile, Bayesian linear models with ReLU features converge, in the infinite-width limit, to a particular Gaussian process (GP) with a variance that grows cubically so that no asymptotic overconfidence can occur. While this may seem of mostly theoretical interest, in this work, we show that it can be used in practice to the benefit of BNNs. We extend finite ReLU BNNs with infinite ReLU features via the GP and show that the resulting model is asymptotically maximally uncertain far away from the data while the BNNs' predictive power is unaffected near the data. Although the resulting model approximates a full GP posterior, thanks to its structure, it can be applied \emph{post-hoc} to any pre-trained ReLU BNN at a low cost.

翻译：贝叶斯治疗可以减轻ReLU网在培训数据周围的过度自信。 但距离它们很远的地方, ReLU Bayesian神经网络(BNNS)仍然可以低估不确定性,从而容易过度自信。 这个问题的出现是因为一个具有有限特性的BNN的输出差异在远离数据区域的地方是二次的。 同时, 带有ReLU特征的巴耶斯线性模型在无限宽度的限度内, 聚集到一个特殊的Gaussian进程(GP), 其差异不断增长, 从而不会发生无症状过度自信。 虽然在这项工作中, 似乎大多具有理论意义, 我们显示它可以在实践中用于BNNUS。 我们通过GP 扩展具有无限ReLU特性的有限 ReLU BNNNNNN, 其结果模型在远离数据时, 极不稳定性极强, 而BNIS的预测力在接近数据附近。 尽管由此产生的模型近于完全的GP posior, 由于其结构低度, 它可以被应用到BHL。