Meeting summarization has emerged as a promising technique for providing users with condensed summaries. However, existing work has focused on training models on centralized data, neglecting real-world scenarios where meeting data are infeasible to collect centrally, due to their sensitive nature. This gap motivates us to explore federated learning for meeting summarization. Two critical challenges impede progress. First, state-of-the-art summarizers are based on parameter-heavy pre-trained models. Exchanging such a model's parameters across clients imposes large bandwidth costs. Second, as real-world meeting data belong to various domains and are distributed across clients, they are instances of non-identically and independently distributed (non-IID). IID assumptions do not hold, which changes which forms of learning algorithms best apply. To address this, we propose Adapter-based Federated Selective Knowledge Distillation (AdaFedSelecKD) for training performant client models. Specifically, we develop an adapter-based summarization model where two adapters cooperatively facilitate learning using fewer parameters to reduce communication costs. Then, we devise a selective knowledge distillation strategy, assisting clients in robustly handling domain-focused modelling on their own data, while leveraging global parameters based on non-IID data. Extensive experiments on the QMSum benchmark demonstrate AdaFedSelecKD can achieve comparable performance with powerful centralized training methods, and shows its generalizability and robustness.
翻译:会议摘要作为一种为用户提供精简摘要的有前景技术已崭露头角。然而,现有工作主要集中于在集中式数据上训练模型,忽略了真实场景中因会议数据具有敏感性而无法集中收集的问题。这一研究空白促使我们探索联邦学习在会议摘要中的应用。两个关键挑战阻碍了进展:首先,最先进的摘要模型基于参数庞大的预训练模型,跨客户端交换此类模型的参数会带来高昂的带宽成本;其次,由于真实会议数据分属不同领域且分布在各客户端,这些数据呈现非独立同分布(non-IID)特征。独立同分布假设不再成立,这改变了最适用的学习算法形式。为此,我们提出基于适配器的联邦选择性知识蒸馏方法(AdaFedSelecKD),用于训练高性能客户端模型。具体而言,我们开发了一种基于适配器的摘要模型,其中两个适配器通过协同使用更少参数促进学习,从而降低通信成本。随后,我们设计了一种选择性知识蒸馏策略,协助客户端基于自身数据稳健地进行领域聚焦建模,同时利用基于非IID数据的全局参数。在QMSum基准上的大量实验表明,AdaFedSelecKD能够达到与强大集中式训练方法相当的性能,并展现出其泛化能力和鲁棒性。