To address the communication burden issues associated with federated learning (FL), decentralized federated learning (DFL) discards the central server and establishes a decentralized communication network, where each client communicates only with neighboring clients. However, existing DFL methods still suffer from two major challenges: local inconsistency and local heterogeneous overfitting, which have not been fundamentally addressed by existing DFL methods. To tackle these issues, we propose novel DFL algorithms, DFedADMM and its enhanced version DFedADMM-SAM, to enhance the performance of DFL. The DFedADMM algorithm employs primal-dual optimization (ADMM) by utilizing dual variables to control the model inconsistency raised from the decentralized heterogeneous data distributions. The DFedADMM-SAM algorithm further improves on DFedADMM by employing a Sharpness-Aware Minimization (SAM) optimizer, which uses gradient perturbations to generate locally flat models and searches for models with uniformly low loss values to mitigate local heterogeneous overfitting. Theoretically, we derive convergence rates of $\small \mathcal{O}\Big(\frac{1}{\sqrt{KT}}+\frac{1}{KT(1-\psi)^2}\Big)$ and $\small \mathcal{O}\Big(\frac{1}{\sqrt{KT}}+\frac{1}{KT(1-\psi)^2}+ \frac{1}{T^{3/2}K^{1/2}}\Big)$ in the non-convex setting for DFedADMM and DFedADMM-SAM, respectively, where $1 - \psi$ represents the spectral gap of the gossip matrix. Empirically, extensive experiments on MNIST, CIFAR10 and CIFAR100 datesets demonstrate that our algorithms exhibit superior performance in terms of both generalization and convergence speed compared to existing state-of-the-art (SOTA) optimizers in DFL.

翻译：为解决联邦学习中的通信负担问题，去中心化联邦学习摒弃中心服务器，构建去中心化通信网络，每个客户端仅与邻近客户端通信。然而，现有去中心化联邦学习方法仍面临两大根本性挑战：局部不一致性与局部异质性过拟合。针对这些问题，我们提出新型去中心化联邦学习算法DFedADMM及其增强版本DFedADMM-SAM，以提升去中心化联邦学习的性能。DFedADMM算法采用原始-对偶优化方法，通过利用对偶变量控制由去中心化异构数据分布导致的模型不一致性问题。DFedADMM-SAM算法进一步引入锐度感知最小化优化器，通过梯度扰动生成局部平坦模型，并搜索具有均匀低损失值的模型以缓解局部异质性过拟合。理论层面，我们在非凸设定下分别推导了DFedADMM与DFedADMM-SAM的收敛率：$\small \mathcal{O}\Big(\frac{1}{\sqrt{KT}}+\frac{1}{KT(1-\psi)^2}\Big)$ 和 $\small \mathcal{O}\Big(\frac{1}{\sqrt{KT}}+\frac{1}{KT(1-\psi)^2}+ \frac{1}{T^{3/2}K^{1/2}}\Big)$，其中$1 - \psi$表示八卦矩阵的谱间隙。实验层面，在MNIST、CIFAR10和CIFAR100数据集上的大量实验表明，相较于现有去中心化联邦学习领域最先进的优化器，本算法在泛化能力与收敛速度方面均展现出卓越性能。