We study methods for estimating model uncertainty for neural networks (NNs) in regression. To isolate the effect of model uncertainty, we focus on a noiseless setting with scarce training data. We introduce five important desiderata regarding model uncertainty that any method should satisfy. However, we find that established benchmarks often fail to reliably capture some of these desiderata, even those that are required by Bayesian theory. To address this, we introduce a new approach for capturing model uncertainty for NNs, which we call Neural Optimization-based Model Uncertainty (NOMU). The main idea of NOMU is to design a network architecture consisting of two connected sub-NNs, one for model prediction and one for model uncertainty, and to train it using a carefully-designed loss function. Importantly, our design enforces that NOMU satisfies our five desiderata. Due to its modular architecture, NOMU can provide model uncertainty for any given (previously trained) NN if given access to its training data. We evaluate NOMU in various regressions tasks and noiseless Bayesian optimization (BO) with costly evaluations. In regression, NOMU performs at least as well as state-of-the-art methods. In BO, NOMU even outperforms all considered benchmarks.

翻译：我们研究了回归任务中神经网络（NN）模型不确定性的估计方法。为分离模型不确定性的影响，我们聚焦于训练数据稀缺且无噪声场景。我们提出了关于模型不确定性的五个重要属性准则，任何方法都应满足这些准则。然而，我们发现现有基准方法往往无法可靠地满足其中部分准则，甚至包括贝叶斯理论所要求的准则。为此，我们提出了一种捕捉神经网络模型不确定性的新方法，称为基于神经优化的模型不确定性（NOMU）。NOMU的核心思想是设计一个由两个连接子神经网络组成的网络架构——一个用于模型预测，另一个用于模型不确定性——并通过精心设计的损失函数进行训练。关键在于，我们的设计确保了NOMU满足上述五个属性准则。由于其模块化架构，若可访问训练数据，NOMU可为任何（预训练）神经网络提供模型不确定性。我们在多种回归任务和代价高昂的无噪声贝叶斯优化（BO）中评估了NOMU。在回归任务中，NOMU表现不逊于最新方法；在贝叶斯优化中，NOMU甚至超越了所有基准方法。