This paper introduces a generalized federated prompt-tuning framework for practical scenarios where local datasets are multi-modal and exhibit different distributional patterns of missing features at the input level. The proposed framework bridges the gap between federated learning and multi-modal prompt-tuning which have traditionally focused on either uni-modal or centralized data. A key challenge in this setting arises from the lack of semantic alignment between prompt instructions that encode similar distributional patterns of missing data across different clients. To address this, our framework introduces specialized client-tuning and server-aggregation designs that simultaneously optimize, align, and aggregate prompt-tuning instructions across clients and data modalities. This allows prompt instructions to complement one another and be combined effectively. Extensive evaluations on diverse multimodal benchmark datasets demonstrate that our work consistently outperforms state-of-the-art (SOTA) baselines.

翻译：本文针对实际场景中本地数据集具有多模态特性且在输入层面呈现不同缺失特征分布模式的情况，提出了一种广义联邦提示调优框架。该框架弥合了联邦学习与多模态提示调优之间的鸿沟——传统方法通常仅关注单模态或集中式数据。在此设定下，一个关键挑战源于不同客户端间编码相似缺失数据分布模式的提示指令缺乏语义对齐。为解决此问题，本框架引入专门的客户端调优与服务器聚合设计，可跨客户端与数据模态同步优化、对齐并聚合提示调优指令。这使得提示指令能够相互补充并有效结合。在多种多模态基准数据集上的广泛实验表明，本方法在各项指标上均持续优于现有最优基线。