This paper focusses on the optimal implementation of a Mean Variance Estimation network (MVE network) (Nix and Weigend, 1994). This type of network is often used as a building block for uncertainty estimation methods in a regression setting, for instance Concrete dropout (Gal et al., 2017) and Deep Ensembles (Lakshminarayanan et al., 2017). Specifically, an MVE network assumes that the data is produced from a normal distribution with a mean function and variance function. The MVE network outputs a mean and variance estimate and optimizes the network parameters by minimizing the negative loglikelihood. In our paper, we present two significant insights. Firstly, the convergence difficulties reported in recent work can be relatively easily prevented by following the simple yet often overlooked recommendation from the original authors that a warm-up period should be used. During this period, only the mean is optimized with a fixed variance. We demonstrate the effectiveness of this step through experimentation, highlighting that it should be standard practice. As a sidenote, we examine whether, after the warm-up, it is beneficial to fix the mean while optimizing the variance or to optimize both simultaneously. Here, we do not observe a substantial difference. Secondly, we introduce a novel improvement of the MVE network: separate regularization of the mean and the variance estimate. We demonstrate, both on toy examples and on a number of benchmark UCI regression data sets, that following the original recommendations and the novel separate regularization can lead to significant improvements.

翻译：本文聚焦于均值方差估计网络（MVE网络）（Nix and Weigend, 1994）的最优实现。这类网络常被用作回归任务中不确定性估计方法的基础模块，例如具体化Dropout（Gal等，2017）和深度集成（Lakshminarayanan等，2017）。具体而言，MVE网络假设数据由包含均值函数和方差函数的正态分布生成，通过输出均值和方差估计值，并以最小化负对数似然为目标优化网络参数。本文提出两项重要见解：其一，近期文献报告的收敛困难可通过遵循原作者简单却常被忽视的建议——使用预热期来相对容易地避免。在此期间，仅优化固定方差下的均值。实验证明该步骤的有效性，我们认为这应成为标准实践。作为补充说明，我们考察了预热后固定均值优化方差与同时优化两者两种方案的优劣，未观察到显著差异。其二，我们提出MVE网络的新改进：对均值和方差估计分别进行正则化。在玩具示例和多个UCI回归基准数据集上的实验表明，遵循原始建议并采用新型分离正则化可带来显著改进。