Traditional federated learning uses the number of samples to calculate the weights of each client model and uses this fixed weight value to fusion the global model. However, in practical scenarios, each client's device and data heterogeneity leads to differences in the quality of each client's model. Thus the contribution to the global model is not wholly determined by the sample size. In addition, if clients intentionally upload low-quality or malicious models, using these models for aggregation will lead to a severe decrease in global model accuracy. Traditional federated learning algorithms do not address these issues. To solve this probelm, we propose FedDRL, a model fusion approach using reinforcement learning based on a two staged approach. In the first stage, Our method could filter out malicious models and selects trusted client models to participate in the model fusion. In the second stage, the FedDRL algorithm adaptively adjusts the weights of the trusted client models and aggregates the optimal global model. We also define five model fusion scenarios and compare our method with two baseline algorithms in those scenarios. The experimental results show that our algorithm has higher reliability than other algorithms while maintaining accuracy.

翻译：传统联邦学习仅依据样本数量计算各客户端模型的权重，并以此固定权重值融合全局模型。然而在实际场景中，各客户端设备与数据的异构性导致模型质量存在差异，因此其对全局模型的贡献并非完全由样本量决定。此外，若客户端故意上传低质量或恶意模型，使用这些模型进行聚合将导致全局模型精度严重下降。传统联邦学习算法未能解决这些问题。为此，我们提出FedDRL——一种基于强化学习的分阶段模型融合方法。在第一阶段，该方法能够过滤恶意模型，筛选可信客户端模型参与模型融合；在第二阶段，FedDRL算法自适应调整可信客户端模型的权重，聚合得到最优全局模型。我们还定义了五种模型融合场景，并在这些场景中将本方法与两种基线算法进行对比。实验结果表明，本算法在保持精度的同时具有更高的可靠性。