While federated learning (FL) systems often utilize quantization to battle communication and computational bottlenecks, they have heretofore been limited to deploying fixed-precision quantization schemes. Meanwhile, the concept of mixed-precision quantization (MPQ), where different layers of a deep learning model are assigned varying bit-width, remains unexplored in the FL settings. We present a novel FL algorithm, FedMPQ, which introduces mixed-precision quantization to resource-heterogeneous FL systems. Specifically, local models, quantized so as to satisfy bit-width constraint, are trained by optimizing an objective function that includes a regularization term which promotes reduction of precision in some of the layers without significant performance degradation. The server collects local model updates, de-quantizes them into full-precision models, and then aggregates them into a global model. To initialize the next round of local training, the server relies on the information learned in the previous training round to customize bit-width assignments of the models delivered to different clients. In extensive benchmarking experiments on several model architectures and different datasets in both iid and non-iid settings, FedMPQ outperformed the baseline FL schemes that utilize fixed-precision quantization while incurring only a minor computational overhead on the participating devices.

翻译：尽管联邦学习系统常利用量化技术应对通信与计算瓶颈，但现有方法始终局限于部署固定精度量化方案。与此同时，混合精度量化——即为深度学习模型不同层分配不同比特宽度的策略——在联邦学习场景中仍属未探索领域。本文提出新型联邦学习算法FedMPQ，将混合精度量化引入资源异构联邦学习系统。具体而言，满足比特宽度约束的局部模型通过优化包含正则化项的目标函数进行训练，该正则化项可在不显著降低性能的前提下促进部分层的精度降低。服务器收集局部模型更新，将其反量化为全精度模型后聚合生成全局模型。为启动下一轮局部训练，服务器基于上一轮训练中习得的信息，为不同客户端的模型定制比特宽度分配方案。在多种模型架构与数据集（涵盖独立同分布与非独立同分布场景）的广泛基准测试中，FedMPQ均优于采用固定精度量化的基线联邦学习方案，且仅在参与设备上引入微小计算开销。