Combining Domain-adaptive Pre-training (DAPT) with Federated Learning (FL) can enhance model adaptation by leveraging more sensitive and distributed data while preserving data privacy. However, few studies have focused on this method. Therefore, we conduct the first comprehensive empirical study to evaluate the performance of Federated Domain-adaptive Pre-training (FDAPT). We demonstrate that FDAPT can maintain competitive downstream task performance to the centralized baseline in both IID and non-IID situations. Furthermore, we propose a novel algorithm, Frozen Federated Domain-adaptive Pre-training (FFDAPT). FFDAPT improves the computational efficiency by 12.1% on average and exhibits similar downstream task performance to standard FDAPT, with general performance fluctuations remaining less than 1%. Finally, through a critical evaluation of our work, we identify promising future research directions for this new research area.

翻译：将域自适应预训练（Domain-adaptive Pre-training, DAPT）与联邦学习（Federated Learning, FL）相结合，可在保护数据隐私的前提下，利用更具敏感性且分布式的数据来增强模型自适应能力。然而，目前鲜有研究聚焦于该方法。为此，我们首次开展全面的实证研究，系统评估联邦域自适应预训练（Federated Domain-adaptive Pre-training, FDAPT）的性能。实验结果表明，无论是在独立同分布（IID）还是非独立同分布（non-IID）场景下，FDAPT均能保持与集中式基线相当的优异下游任务性能。此外，我们提出一种新型算法——冻结式联邦域自适应预训练（Frozen Federated Domain-adaptive Pre-training, FFDAPT）。FFDAPT将计算效率平均提升12.1%，同时展现出与标准FDAPT相近的下游任务性能，整体性能波动幅度低于1%。最后，通过对本研究的批判性评估，我们为这一新兴研究领域指明了具有前景的未来研究方向。