As AI becomes more personal, e.g., Agentic AI, there is an increasing need for personalizing models for various use cases. Personalized federated learning (PFL) enables each client to collaboratively leverage other clients' knowledge for better adaptation to the task of interest, without privacy risks. Despite its potential, existing PFL methods remain confined to rather simplified scenarios where data and models are the same across clients. To move towards realistic scenarios, we move beyond these restrictive assumptions by addressing both data and model heterogeneity. We propose a task-relevance-aware model aggregation strategy to reduce parameter interference under heterogeneous data. Moreover, we introduce Co-LoRA, a dimension-invariant module that enables knowledge sharing across heterogeneous architectures. To mimic the real-world task diversity, we propose a multi-modal PFL benchmark spanning 40 distinct tasks with distribution shifts over time. Extensive experiments shows that our proposed method significantly outperforms the state-of-the-art PFL methods under heterogeneous scenarios.

翻译：随着人工智能日益趋向个性化（例如智能体AI），针对多样化用例进行模型个性化的需求日益增长。个性化联邦学习（PFL）使得每个客户端能够协同利用其他客户端的知识，以更好地适应目标任务，同时规避隐私风险。尽管潜力巨大，现有PFL方法仍局限于客户端间数据和模型完全相同的简化场景。为迈向现实场景，我们突破这些限制性假设，同时处理数据与模型异构性问题。我们提出一种任务相关性感知的模型聚合策略，以降低异构数据下的参数干扰。此外，我们引入Co-LoRA——一种维度无关的模块，支持跨异构架构的知识共享。为模拟现实世界的任务多样性，我们构建了一个涵盖40个具有时序分布偏移任务的跨模态PFL基准测试集。大量实验表明，在异构场景下，我们提出的方法显著优于当前最先进的PFL方法。