In federated learning, data heterogeneity significantly impacts performance. A typical solution involves segregating these parameters into shared and personalized components, a concept also relevant in multi-task learning. Addressing this, we propose "Loop Improvement" (LI), a novel method enhancing this separation and feature extraction without necessitating a central server or data interchange among participants. Our experiments reveal LI's superiority in several aspects: In personalized federated learning environments, LI consistently outperforms the advanced FedALA algorithm in accuracy across diverse scenarios. Additionally, LI's feature extractor closely matches the performance achieved when aggregating data from all clients. In global model contexts, employing LI with stacked personalized layers and an additional network also yields comparable results to combined client data scenarios. Furthermore, LI's adaptability extends to multi-task learning, streamlining the extraction of common features across tasks and obviating the need for simultaneous training. This approach not only enhances individual task performance but also achieves accuracy levels on par with classic multi-task learning methods where all tasks are trained simultaneously. LI integrates a loop topology with layer-wise and end-to-end training, compatible with various neural network models. This paper also delves into the theoretical underpinnings of LI's effectiveness, offering insights into its potential applications. The code is on https://github.com/axedge1983/LI

翻译：在联邦学习中，数据异构性显著影响性能。典型的解决方案是将这些参数分解为共享组件和个性化组件，这一概念在多任务学习中同样适用。针对这一问题，我们提出了“循环改进”（Loop Improvement, LI）这一新颖方法，在无需中心服务器或参与方间数据交换的情况下，增强了这种参数分离与特征提取能力。实验表明，LI在多个方面具有优越性：在个性化联邦学习环境中，LI在不同场景下的准确率始终优于先进的FedALA算法。同时，LI的特征提取器在性能上接近聚合所有客户端数据时的表现。在全局模型场景中，将LI与堆叠的个性化层及附加网络结合使用，也能取得与合并客户端数据场景相当的结果。此外，LI的适应性可扩展至多任务学习，能简化跨任务公共特征的提取过程，且无需同步训练。这种方法不仅提升了单个任务性能，还能达到与所有任务同步训练的经典多任务学习方法相当的准确率。LI集成了循环拓扑结构与逐层及端到端训练机制，兼容多种神经网络模型。本文还深入探讨了LI有效性的理论依据，并对其潜在应用提供了见解。代码参见https://github.com/axedge1983/LI