Federated Learning (FL) has been an area of active research in recent years. There have been numerous studies in FL to make it more successful in the presence of data heterogeneity. However, despite the existence of many publications, the state of progress in the field is unknown. Many of the works use inconsistent experimental settings and there are no comprehensive studies on the effect of FL-specific experimental variables on the results and practical insights for a more comparable and consistent FL experimental setup. Furthermore, the existence of several benchmarks and confounding variables has further complicated the issue of inconsistency and ambiguity. In this work, we present the first comprehensive study on the effect of FL-specific experimental variables in relation to each other and performance results, bringing several insights and recommendations for designing a meaningful and well-incentivized FL experimental setup. We further aid the community by releasing FedZoo-Bench, an open-source library based on PyTorch with pre-implementation of 22 state-of-the-art methods, and a broad set of standardized and customizable features available at https://github.com/MMorafah/FedZoo-Bench. We also provide a comprehensive comparison of several state-of-the-art (SOTA) methods to better understand the current state of the field and existing limitations.

翻译：联邦学习（FL）是近年来活跃的研究领域。针对数据异构性问题，已有大量研究致力于提升FL性能。然而，尽管相关出版物众多，该领域的研究进展仍不明朗。许多工作采用不一致的实验设置，且缺乏对FL特定实验变量如何影响结果的系统性研究，以及针对更可比、更一致的FL实验设置的实际指导。此外，多个基准测试集与混杂变量的存在进一步加剧了实验设置不一致与结果模糊的问题。本文首次系统研究了FL特定实验变量之间的相互关联及其对性能结果的影响，提出了若干关于设计有意义且激励充分的FL实验设置的见解与建议。我们进一步发布了FedZoo-Bench开源库（基于PyTorch，预实现了22种最新方法，并提供标准化与可定制化功能集，访问地址：https://github.com/MMorafah/FedZoo-Bench），为社区提供支持。同时，我们全面比较了多种最新方法，以更清晰揭示该领域的研究现状与现有局限。