The innovative Federated Multi-Task Learning (FMTL) approach consolidates the benefits of Federated Learning (FL) and Multi-Task Learning (MTL), enabling collaborative model training on multi-task learning datasets. However, a comprehensive evaluation method, integrating the unique features of both FL and MTL, is currently absent in the field. This paper fills this void by introducing a novel framework, FMTL-Bench, for systematic evaluation of the FMTL paradigm. This benchmark covers various aspects at the data, model, and optimization algorithm levels, and comprises seven sets of comparative experiments, encapsulating a wide array of non-independent and identically distributed (Non-IID) data partitioning scenarios. We propose a systematic process for comparing baselines of diverse indicators and conduct a case study on communication expenditure, time, and energy consumption. Through our exhaustive experiments, we aim to provide valuable insights into the strengths and limitations of existing baseline methods, contributing to the ongoing discourse on optimal FMTL application in practical scenarios. The source code will be made available for results replication.

翻译：创新的联邦多任务学习方法融合了联邦学习和多任务学习的优势，能够在多任务学习数据集上实现协同模型训练。然而，目前该领域尚缺乏一种融合FL与MTL独特特性的综合评估方法。本文通过引入系统性评估FMTL范式的新型框架FMTL-Bench填补了这一空白。该基准从数据层面、模型层面及优化算法层面覆盖多个维度，并包含七组对比实验，囊括了广泛的非独立同分布数据划分场景。我们提出了一种用于对比多指标基线的系统流程，并针对通信开销、时间与能耗进行了案例研究。通过详尽的实验，我们旨在为现有基线方法的优势与局限性提供有价值的见解，为实际场景中FMTL最优应用的相关持续讨论做出贡献。源代码将公开以供结果复现。