Bayesian model reduction provides an efficient approach for comparing the performance of all nested sub-models of a model, without re-evaluating any of these sub-models. Until now, Bayesian model reduction has been applied mainly in the computational neuroscience community on simple models. In this paper, we formulate and apply Bayesian model reduction to perform principled pruning of Bayesian neural networks, based on variational free energy minimization. Direct application of Bayesian model reduction, however, gives rise to approximation errors. Therefore, a novel iterative pruning algorithm is presented to alleviate the problems arising with naive Bayesian model reduction, as supported experimentally on the publicly available UCI datasets for different inference algorithms. This novel parameter pruning scheme solves the shortcomings of current state-of-the-art pruning methods that are used by the signal processing community. The proposed approach has a clear stopping criterion and minimizes the same objective that is used during training. Next to these benefits, our experiments indicate better model performance in comparison to state-of-the-art pruning schemes.

翻译：贝叶斯模型约减提供了一种高效方法，用于比较模型所有嵌套子模型的性能，而无需对这些子模型进行重新评估。此前，贝叶斯模型约减主要被计算神经科学社区应用于简单模型。本文基于变分自由能最小化，将贝叶斯模型约减方法进行公式化并应用于贝叶斯神经网络的原理性剪枝。然而，直接应用贝叶斯模型约减会产生近似误差。为此，我们提出一种新颖的迭代剪枝算法，以缓解朴素贝叶斯模型约减所引发的问题。不同推理算法在公开UCI数据集上的实验验证了该方法的有效性。这种新颖的参数剪枝方案解决了当前信号处理领域最先进剪枝方法的不足。所提方法具有明确的停止准则，且优化目标与训练阶段保持一致。除上述优势外，实验表明与现有最优剪枝方案相比，本方法能实现更优的模型性能。