Foundation models (FMs) have shown prominent success in a wide range of tasks. Their applicability to specific domain-task pairings relies on the availability of, both, high-quality data and significant computational resources. These challenges are not new to the field and, indeed, Federated Learning (FL) has been shown to be a promising solution in similar setups. This paper tackles the specific case of Domain-Adaptive Pre-Training (DAPT), a key step in the application of FMs. We conduct the first comprehensive empirical study to evaluate the performance of Federated Domain-Adaptive Pre-Training (FDAPT). We demonstrate that FDAPT can maintain competitive downstream task performance to the centralized baseline in both IID and non-IID situations. Finally, we propose a novel algorithm, Frozen Federated Domain-Adaptive Pre-Training (FFDAPT). FFDAPT improves the computational efficiency by 12.1% on average and exhibits similar downstream task performance to vanilla FDAPT, with general performance fluctuations remaining less than 1%.

翻译：基础模型（FMs）在广泛任务中展现出显著成功。其针对特定领域-任务配对的可应用性依赖于高质量数据和充足计算资源的双重可用性。这些挑战对领域而言并不陌生，事实上，联邦学习（FL）已在类似场景中被证明是一种有前景的解决方案。本文针对域自适应预训练（DAPT）这一基础模型应用中的关键步骤展开研究。我们首次开展了全面的实证研究，以评估联邦域自适应预训练（FDAPT）的性能。实验证明，在独立同分布（IID）和非独立同分布（non-IID）场景下，FDAPT均能保持与集中式基线相当的后续任务性能。最后，我们提出了一种新型算法——冻结联邦域自适应预训练（FFDAPT）。该算法平均计算效率提升12.1%，且下游任务性能与标准FDAPT相近，整体性能波动低于1%。