We derive an (almost) guaranteed upper bound on the error of deep neural networks under distribution shift using unlabeled test data. Prior methods either give bounds that are vacuous in practice or give estimates that are accurate on average but heavily underestimate error for a sizeable fraction of shifts. In particular, the latter only give guarantees based on complex continuous measures such as test calibration -- which cannot be identified without labels -- and are therefore unreliable. Instead, our bound requires a simple, intuitive condition which is well justified by prior empirical works and holds in practice effectively 100% of the time. The bound is inspired by $\mathcal{H}\Delta\mathcal{H}$-divergence but is easier to evaluate and substantially tighter, consistently providing non-vacuous guarantees. Estimating the bound requires optimizing one multiclass classifier to disagree with another, for which some prior works have used sub-optimal proxy losses; we devise a "disagreement loss" which is theoretically justified and performs better in practice. We expect this loss can serve as a drop-in replacement for future methods which require maximizing multiclass disagreement. Across a wide range of benchmarks, our method gives valid error bounds while achieving average accuracy comparable to competitive estimation baselines. Code is publicly available at https://github.com/erosenfeld/disagree_discrep .

翻译：我们利用无标签测试数据推导出深度神经网络在分布漂移下的（几乎）有保证的误差上界。先前的方法要么在实践中给出空洞的界，要么给出平均准确但会严重低估相当一部分漂移误差的估计。特别地，后者仅基于测试校准等复杂连续度量提供保证——这些度量在无标签时无法识别——因此不可靠。相比之下，我们的界仅需一个简单直观的条件，该条件得到先前实证工作的充分支持，并且在实践中几乎100%有效。该界受$\mathcal{H}\Delta\mathcal{H}$散度启发，但更易评估且显著更紧，始终能提供非空洞的保证。估计该界需要优化一个多类分类器使其与另一个分类器产生分歧，现有工作对此使用了次优代理损失；我们设计了一种理论上合理且在实践中表现更优的"分歧损失"。我们预期该损失可作为未来需要最大化多类分歧方法的即插即用替代方案。在广泛基准测试中，我们的方法在保持与竞争性估计基线相当的平均精度的同时，给出了有效的误差界。代码已公开于 https://github.com/erosenfeld/disagree_discrep。