Traditional federated learning uses the number of samples to calculate the weights of each client model and uses this fixed weight value to fusion the global model. However, in practical scenarios, each client's device and data heterogeneity leads to differences in the quality of each client's model. Thus the contribution to the global model is not wholly determined by the sample size. In addition, if clients intentionally upload low-quality or malicious models, using these models for aggregation will lead to a severe decrease in global model accuracy. Traditional federated learning algorithms do not address these issues. To solve this probelm, we propose FedDRL, a model fusion approach using reinforcement learning based on a two staged approach. In the first stage, Our method could filter out malicious models and selects trusted client models to participate in the model fusion. In the second stage, the FedDRL algorithm adaptively adjusts the weights of the trusted client models and aggregates the optimal global model. We also define five model fusion scenarios and compare our method with two baseline algorithms in those scenarios. The experimental results show that our algorithm has higher reliability than other algorithms while maintaining accuracy.

翻译：传统联邦学习使用样本数量计算各客户端模型的权重，并以此固定权重值进行全局模型融合。然而实际场景中，各客户端设备与数据的异质性导致客户端模型质量存在差异，其对全局模型的贡献并非完全由样本量决定。此外，若客户端故意上传低质量或恶意模型，使用这些模型进行聚合将导致全局模型精度严重下降。传统联邦学习算法未解决这些问题。针对此问题，本文提出FedDRL——一种基于强化学习的两阶段模型融合方法。第一阶段可过滤恶意模型，筛选可信客户端模型参与融合；第二阶段自适应调整可信客户端模型权重，聚合最优全局模型。同时定义五种模型融合场景，将所提方法与两种基线算法进行对比。实验结果表明，本算法在保持精度的同时具有更高的可靠性。