The growth and diversity of machine learning applications motivate a rethinking of learning with mobile and edge devices. How can we address diverse client goals and learn with scarce heterogeneous data? While federated learning aims to address these issues, it has challenges hindering a unified solution. Large transformer models have been shown to work across a variety of tasks achieving remarkable few-shot adaptation. This raises the question: Can clients use a single general-purpose model, rather than custom models for each task, while obeying device and network constraints? In this work, we investigate pretrained transformers (PTF) to achieve these on-device learning goals and thoroughly explore the roles of model size and modularity, where the latter refers to adaptation through modules such as prompts or adapters. Focusing on federated learning, we demonstrate that: (1) Larger scale shrinks the accuracy gaps between alternative approaches and improves heterogeneity robustness. Scale allows clients to run more local SGD epochs which can significantly reduce the number of communication rounds. At the extreme, clients can achieve respectable accuracy locally highlighting the potential of fully-local learning. (2) Modularity, by design, enables $>$100$\times$ less communication in bits. Surprisingly, it also boosts the generalization capability of local adaptation methods and the robustness of smaller PTFs. Finally, it enables clients to solve multiple unrelated tasks simultaneously using a single PTF, whereas full updates are prone to catastrophic forgetting. These insights on scale and modularity motivate a new federated learning approach we call "You Only Load Once" (FedYolo): The clients load a full PTF model once and all future updates are accomplished through communication-efficient modules with limited catastrophic-forgetting, where each task is assigned to its own module.

翻译：机器学习应用的增长与多样性促使我们重新思考移动与边缘设备上的学习方式。如何应对多样化的客户端目标，并在稀缺的异构数据中进行学习？尽管联邦学习旨在解决这些问题，但当前仍存在阻碍统一解决方案的挑战。大型Transformer模型已被证明可跨多种任务工作，并展现出卓越的少样本自适应能力。这引发了一个问题：在遵守设备和网络约束的同时，客户端能否使用单一通用模型而非为每项任务定制模型？本研究探索利用预训练Transformer（PTF）实现这些设备端学习目标，并深入分析模型规模与模块性的作用——后者指通过提示（prompt）或适配器（adapter）等模块进行自适应。聚焦联邦学习，我们证明：(1) 更大规模可缩小各方法间的精度差距并提升异构鲁棒性。规模增长使客户端能执行更多本地SGD轮次，从而显著减少通信轮数。极端情况下，客户端可在本地达到可观精度，彰显了全本地学习的潜力。(2) 模块性设计可实现超过100倍的比特通信量降低。令人惊讶的是，它还能提升本地自适应方法的泛化能力以及较小PTF的鲁棒性。最终，客户端可使用单个PTF同时解决多个无关任务，而全参数更新容易引发灾难性遗忘。这些关于规模与模块性的洞察催生了一种新的联邦学习方法——"单次加载即可"（FedYolo）：客户端仅需完整加载一次PTF模型，所有后续更新均通过通信高效的模块完成，且各模块仅承载单一任务，显著降低灾难性遗忘风险。