Bayesian neural network (BNN) allows for uncertainty quantification in prediction, offering an advantage over regular neural networks that has not been explored in the differential privacy (DP) framework. We fill this important gap by leveraging recent development in Bayesian deep learning and privacy accounting to offer a more precise analysis of the trade-off between privacy and accuracy in BNN. We propose three DP-BNNs that characterize the weight uncertainty for the same network architecture in distinct ways, namely DP-SGLD (via the noisy gradient method), DP-BBP (via changing the parameters of interest) and DP-MC Dropout (via the model architecture). Interestingly, we show a new equivalence between DP-SGD and DP-SGLD, implying that some non-Bayesian DP training naturally allows for uncertainty quantification. However, the hyperparameters such as learning rate and batch size, can have different or even opposite effects in DP-SGD and DP-SGLD. Extensive experiments are conducted to compare DP-BNNs, in terms of privacy guarantee, prediction accuracy, uncertainty quantification, calibration, computation speed, and generalizability to network architecture. As a result, we observe a new tradeoff between the privacy and the reliability. When compared to non-DP and non-Bayesian approaches, DP-SGLD is remarkably accurate under strong privacy guarantee, demonstrating the great potential of DP-BNN in real-world tasks.

翻译：贝叶斯神经网络（BNN）能够实现预测中的不确定性量化，这一特性优于常规神经网络，但在差分隐私（DP）框架中尚未得到探索。我们通过利用贝叶斯深度学习和隐私核算的最新进展来填补这一重要空白，从而更精确地分析BNN中隐私与准确性之间的权衡。我们提出了三种DP-BNN模型，它们以不同方式表征相同网络架构的权重不确定性，即DP-SGLD（通过噪声梯度方法）、DP-BBP（通过改变感兴趣参数）和DP-MC Dropout（通过模型架构）。有趣的是，我们展示了DP-SGD与DP-SGLD之间的一种新等价关系，这表明某些非贝叶斯DP训练自然允许不确定性量化。然而，超参数（如学习率和批量大小）在DP-SGD和DP-SGLD中可能产生不同甚至相反的影响。我们进行了大量实验，从隐私保证、预测准确性、不确定性量化、校准、计算速度以及对网络架构的泛化能力等方面比较了DP-BNN。结果发现，隐私与可靠性之间存在一种新的权衡关系。与非DP和非贝叶斯方法相比，DP-SGLD在强隐私保证下仍具有显著准确性，展示了DP-BNN在实际任务中的巨大潜力。