Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. However, the performance of the global model is often hampered by non-i.i.d. distribution among the clients, requiring extensive efforts to mitigate inter-client data heterogeneity. Going beyond inter-client data heterogeneity, we note that intra-client heterogeneity can also be observed on complex real-world data and seriously deteriorate FL performance. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Instead of huge instance-adaptive models, we resort to a parameter-efficient fine-tuning method, i.e., scale and shift deep features (SSF), upon a pre-trained model. Specifically, we first train an SSF pool for each client, and aggregate these SSF pools on the server side, thus still maintaining a low communication cost. To enable instance-adaptive inference, for a given instance, we dynamically find the best-matched SSF subsets from the pool and aggregate them to generate an adaptive SSF specified for the instance, thereby reducing the intra-client as well as the inter-client heterogeneity. Extensive experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64\% improvement against the top-performing method with less than 15\% communication cost on Tiny-ImageNet. Our code and models will be publicly released.

翻译：联邦学习（FL）是一种分布式学习范式，通过聚合本地训练使多个客户端能够学习强大的全局模型。然而，全局模型的性能常受客户端间非独立同分布（non-i.i.d.）数据分布的制约，需投入大量努力来缓解客户端间的数据异质性。在关注客户端间数据异质性的基础上，我们注意到复杂现实数据中同样存在客户端内数据异质性，这会严重恶化联邦学习性能。本文提出一种新型联邦学习算法FedIns，通过在联邦学习框架中实现实例自适应推理来处理客户端内数据异质性。不同于构建庞大的实例自适应模型，我们采用参数高效微调方法——即深层特征缩放与偏移（SSF）技术——基于预训练模型进行优化。具体而言，我们首先为每个客户端训练SSF池，并在服务器端对这些SSF池进行聚合，从而保持低通信成本。为实现实例自适应推理，对于给定实例，我们从池中动态寻找最匹配的SSF子集并聚合生成实例专属的自适应SSF，从而同时降低客户端内和客户端间异质性。大量实验表明，我们的FedIns优于当前最先进的联邦学习算法，例如在Tiny-ImageNet数据集上，相较于最优方法，仅需不到15%的通信成本即可实现6.64%的性能提升。我们的代码和模型将公开发布。