We study how to train personalized models for different tasks on decentralized devices with limited local data. We propose "Structured Cooperative Learning (SCooL)", in which a cooperation graph across devices is generated by a graphical model prior to automatically coordinate mutual learning between devices. By choosing graphical models enforcing different structures, we can derive a rich class of existing and novel decentralized learning algorithms via variational inference. In particular, we show three instantiations of SCooL that adopt Dirac distribution, stochastic block model (SBM), and attention as the prior generating cooperation graphs. These EM-type algorithms alternate between updating the cooperation graph and cooperative learning of local models. They can automatically capture the cross-task correlations among devices by only monitoring their model updating in order to optimize the cooperation graph. We evaluate SCooL and compare it with existing decentralized learning methods on an extensive set of benchmarks, on which SCooL always achieves the highest accuracy of personalized models and significantly outperforms other baselines on communication efficiency. Our code is available at https://github.com/ShuangtongLi/SCooL.

翻译：我们研究如何在数据有限的分散设备上为不同任务训练个性化模型。本文提出"结构化协作学习(SCooL)"方法，通过图模型先验自动生成设备间的协作图来协调设备间的相互学习。通过选择不同结构的图模型，可借助变分推断衍生出丰富多样的现有及新型去中心化学习算法。具体而言，我们展示了SCooL的三种实例化方案，分别采用狄拉克分布、随机块模型(SBM)和注意力机制作为生成协作图的先验。这些EM型算法交替更新协作图与局部模型的协作学习，仅通过监测模型更新过程即可自动捕捉设备间的跨任务相关性以优化协作图。我们在广泛基准测试上评估了SCooL并将其与现有去中心化学习方法进行对比，结果显示SCooL始终取得最高的个性化模型准确率，并在通信效率上显著优于其他基线方法。我们的代码开源于 https://github.com/ShuangtongLi/SCooL。