Federated Multimodal Learning (FML) trains multimodal models across decentralized clients while keeping their image-text pairs private. However, joint embedding training entangles forgotten knowledge across both modalities and client gradient subspaces, hindering federated unlearning. Previous federated unlearning approaches neither sever the cross-modal reconstruction channel mediated by bilinear coupling nor separate forget-exclusive update directions from those shared with retained clients. We identify an Anchor Principle for federated multimodal contrastive unlearning: forgotten alignments persist through three residual anchors arising from bilinear cross-modal coupling, principal-angle subspace entanglement, and continued federated updates. At the modality level, we show that bilateral displacement of both visual and language branches closes the cross-modal reconstruction channel. Correspondingly, our method addresses subspace entanglement through Cosine--Sine decomposition of client-update subspaces, isolating forget-exclusive directions from retain support. Moreover, we propose a direction-selective Forget Lock that bounds residual drift across rounds. Combining these strategies, we present EASE, an Entanglement-Aware Subspace Excision framework that closes all three anchor channels under a unified design. EASE demonstrates consistent superiority across multiple datasets and unlearning scenarios, for instance, matching the retrain reference to within 0.2 and 4.2 R@1 points on the forget and retain sides under client unlearning on Flickr30K with CLIP-B/32.
翻译:联邦多模态学习(FML)在分布式客户端上训练多模态模型,同时保持其图像-文本对的隐私性。然而,联合嵌入训练会将遗忘的知识纠缠到两种模态和客户端梯度子空间中,从而阻碍联邦遗忘。以往的联邦遗忘方法既无法切断由双线性耦合介导的跨模态重构通道,也无法将从遗忘样本独有的更新方向与保留样本共享的更新方向分离。我们为联邦多模态对比遗忘发现了一个锚点原理:遗忘的对齐关系通过三个残差锚点持续存在,这些锚点源于双线性跨模态耦合、主角子空间纠缠以及持续的联邦更新。在模态层面,我们证明对视觉和语言分支的双侧位移能够关闭跨模态重构通道。相应地,我们的方法通过客户端更新子空间的余弦-正弦分解来解决子空间纠缠问题,将遗忘专属方向与保留支持方向分离。此外,我们提出了一种方向选择性的"遗忘锁",用于约束跨轮次的残差漂移。结合这些策略,我们提出了EASE——一个基于纠缠感知的子空间切除框架,在统一设计下关闭所有三个锚点通道。EASE在多个数据集和遗忘场景中展现出持续的优势,例如在使用CLIP-B/32对Flickr30K进行客户端遗忘时,遗忘侧和保留侧的R@1指标与重训练基准的差距分别不超过0.2和4.2个百分点。