Bayesian neural networks (BNNs) have recently regained a significant amount of attention in the deep learning community due to the development of scalable approximate Bayesian inference techniques. There are several advantages of using a Bayesian approach: Parameter and prediction uncertainties become easily available, facilitating rigorous statistical analysis. Furthermore, prior knowledge can be incorporated. However, so far, there have been no scalable techniques capable of combining both structural and parameter uncertainty. In this paper, we apply the concept of model uncertainty as a framework for structural learning in BNNs and hence make inference in the joint space of structures/models and parameters. Moreover, we suggest an adaptation of a scalable variational inference approach with reparametrization of marginal inclusion probabilities to incorporate the model space constraints. Experimental results on a range of benchmark datasets show that we obtain comparable accuracy results with the competing models, but based on methods that are much more sparse than ordinary BNNs.

翻译：贝叶斯神经网络（BNNs）近年来因可扩展近似贝叶斯推断技术的发展，在深度学习领域重新获得了显著关注。采用贝叶斯方法具有若干优势：参数与预测不确定性易于获取，从而促进严谨的统计分析；此外，先验知识也可被纳入。然而，目前尚无可扩展技术能够同时整合结构与参数不确定性。本文中，我们将模型不确定性概念作为BNNs结构学习的框架，从而在结构/模型与参数的联合空间中进行推断。同时，我们提出了一种自适应的可扩展变分推断方法，通过重新参数化边际包含概率来纳入模型空间约束。在多个基准数据集上的实验结果表明，我们获得了与竞争模型相当的准确率，但所采用的方法比普通BNNs更为稀疏。