The innovative Federated Multi-Task Learning (FMTL) approach consolidates the benefits of Federated Learning (FL) and Multi-Task Learning (MTL), enabling collaborative model training on multi-task learning datasets. However, a comprehensive evaluation method, integrating the unique features of both FL and MTL, is currently absent in the field. This paper fills this void by introducing a novel framework, FMTL-Bench, for systematic evaluation of the FMTL paradigm. This benchmark covers various aspects at the data, model, and optimization algorithm levels, and comprises seven sets of comparative experiments, encapsulating a wide array of non-independent and identically distributed (Non-IID) data partitioning scenarios. We propose a systematic process for comparing baselines of diverse indicators and conduct a case study on communication expenditure, time, and energy consumption. Through our exhaustive experiments, we aim to provide valuable insights into the strengths and limitations of existing baseline methods, contributing to the ongoing discourse on optimal FMTL application in practical scenarios. The source code can be found on https://github.com/youngfish42/FMTL-Benchmark .

翻译：创新的联邦多任务学习（FMTL）方法融合了联邦学习（FL）与多任务学习（MTL）的优势，使得在多任务学习数据集上实现协作式模型训练成为可能。然而，当前领域内尚缺乏一种能够整合FL与MTL独特特征的综合评估方法。本文通过引入一种新的系统评估框架FMTL-Bench，填补了这一空白。该基准覆盖了数据、模型及优化算法层面的多个维度，并包含七组对比实验，涵盖了广泛的非独立同分布（Non-IID）数据划分场景。我们提出了一套系统化的流程，用于对比不同指标下的基线方法，并针对通信开销、时间与能耗进行了案例研究。通过详尽的实验，我们旨在揭示现有基线方法的优势与局限性，为实际场景中FMTL的最优应用提供宝贵见解。源代码可访问 https://github.com/youngfish42/FMTL-Benchmark 获取。