We study how to train personalized models for different tasks on decentralized devices with limited local data. We propose "Structured Cooperative Learning (SCooL)", in which a cooperation graph across devices is generated by a graphical model prior to automatically coordinate mutual learning between devices. By choosing graphical models enforcing different structures, we can derive a rich class of existing and novel decentralized learning algorithms via variational inference. In particular, we show three instantiations of SCooL that adopt Dirac distribution, stochastic block model (SBM), and attention as the prior generating cooperation graphs. These EM-type algorithms alternate between updating the cooperation graph and cooperative learning of local models. They can automatically capture the cross-task correlations among devices by only monitoring their model updating in order to optimize the cooperation graph. We evaluate SCooL and compare it with existing decentralized learning methods on an extensive set of benchmarks, on which SCooL always achieves the highest accuracy of personalized models and significantly outperforms other baselines on communication efficiency. Our code is available at https://github.com/ShuangtongLi/SCooL.

翻译：我们研究了在本地数据有限的分散设备上，针对不同任务训练个性化模型的方法。提出"结构化协作学习（SCooL）"框架，该框架通过图模型先验自动生成设备间的协作图，以协调设备间的相互学习。通过选择具有不同结构约束的图模型，可以基于变分推理导出一系列现有及新型的分散式学习算法。特别地，我们展示了SCooL的三种具体实现：分别采用狄拉克分布、随机块模型（SBM）和注意力机制作为协作图的生成先验。这些EM类算法通过交替更新协作图与本地模型的协同学习，仅通过监控模型更新过程即可自动捕捉设备间的跨任务相关性，从而优化协作图结构。我们在广泛基准测试上评估了SCooL，并与现有分散式学习方法进行对比。实验结果表明，SCooL在个性化模型精确度上始终达到最优水平，同时在通信效率上显著优于其他基线方法。我们的代码已开源至https://github.com/ShuangtongLi/SCooL。