Federated learning (FL) is an approach to training machine learning models that takes advantage of multiple distributed datasets while maintaining data privacy and reducing communication costs associated with sharing local datasets. Aggregation strategies have been developed to pool or fuse the weights and biases of distributed deterministic models; however, modern deterministic deep learning (DL) models are often poorly calibrated and lack the ability to communicate a measure of epistemic uncertainty in prediction, which is desirable for remote sensing platforms and safety-critical applications. Conversely, Bayesian DL models are often well calibrated and capable of quantifying and communicating a measure of epistemic uncertainty along with a competitive prediction accuracy. Unfortunately, because the weights and biases in Bayesian DL models are defined by a probability distribution, simple application of the aggregation methods associated with FL schemes for deterministic models is either impossible or results in sub-optimal performance. In this work, we use independent and identically distributed (IID) and non-IID partitions of the CIFAR-10 dataset and a fully variational ResNet-20 architecture to analyze six different aggregation strategies for Bayesian DL models. Additionally, we analyze the traditional federated averaging approach applied to an approximate Bayesian Monte Carlo dropout model as a lightweight alternative to more complex variational inference methods in FL. We show that aggregation strategy is a key hyperparameter in the design of a Bayesian FL system with downstream effects on accuracy, calibration, uncertainty quantification, training stability, and client compute requirements.

翻译：联邦学习是一种训练机器学习模型的方法，它利用多个分布式数据集的同时，既能维护数据隐私，又能降低共享本地数据集所需的通信成本。目前已开发出多种聚合策略，用于合并或融合分布式确定性模型的权重和偏置。然而，现代确定性深度学习模型通常校准不佳，且无法在预测中传达认知不确定性度量，这在遥感平台和安全关键应用中尤为重要。相反，贝叶斯深度学习模型往往校准良好，能够量化和传达认知不确定性度量，同时具有具有竞争力的预测精度。但问题在于，由于贝叶斯深度学习模型中的权重和偏置由概率分布定义，简单应用联邦学习方案中针对确定性模型的聚合方法要么行不通，要么导致性能欠佳。本研究利用CIFAR-10数据集的独立同分布和非独立同分布划分，以及全变分ResNet-20架构，分析了六种针对贝叶斯深度学习模型的聚合策略。此外，我们将传统联邦平均方法应用于近似贝叶斯蒙特卡洛丢弃模型，作为联邦学习中更复杂变分推断方法的轻量级替代方案。我们证明，聚合策略是设计贝叶斯联邦学习系统的关键超参数，对精度、校准、不确定性量化、训练稳定性以及客户端计算需求均会产生下游影响。