As data size and computing power increase, the architectures of deep neural networks (DNNs) have been getting more complex and huge, and thus there is a growing need to simplify such complex and huge DNNs. In this paper, we propose a novel sparse Bayesian neural network (BNN) which searches a good DNN with an appropriate complexity. We employ the masking variables at each node which can turn off some nodes according to the posterior distribution to yield a nodewise sparse DNN. We devise a prior distribution such that the posterior distribution has theoretical optimalities (i.e. minimax optimality and adaptiveness), and develop an efficient MCMC algorithm. By analyzing several benchmark datasets, we illustrate that the proposed BNN performs well compared to other existing methods in the sense that it discovers well condensed DNN architectures with similar prediction accuracy and uncertainty quantification compared to large DNNs.

翻译：随着数据规模和计算能力的增长，深度神经网络的结构日益复杂庞大，因此简化此类复杂庞大的深度神经网络的需求日益迫切。本文提出一种新型稀疏贝叶斯神经网络，能够搜索具有适当复杂度的优秀深度神经网络。我们在每个节点引入掩码变量，根据后验分布关闭部分节点，从而生成节点级稀疏深度神经网络。我们设计了先验分布，使后验分布具备理论最优性（即极小化极大最优性与自适应能力），并开发了高效的MCMC算法。通过分析多个基准数据集，我们发现相较于其他现有方法，本文提出的贝叶斯神经网络能够发现结构高度精简的深度神经网络架构，且在预测精度与不确定性量化方面与大型深度神经网络表现相当。