Although distributed machine learning (distributed ML) is gaining considerable attention in the community, prior works have independently looked at instances of distributed ML in either the training or the inference phase. No prior work has examined the combined robustness stemming from distributing both the learning and the inference process. In this work, we explore, for the first time, the robustness of distributed ML models that are fully heterogeneous in training data, architecture, scheduler, optimizer, and other model parameters. Supported by theory and extensive experimental validation using CIFAR10 and FashionMNIST, we show that such properly distributed ML instantiations achieve across-the-board improvements in accuracy-robustness tradeoffs against state-of-the-art transfer-based attacks that could otherwise not be realized by current ensemble or federated learning instantiations. For instance, our experiments on CIFAR10 show that for the Common Weakness attack, one of the most powerful state-of-the-art transfer-based attacks, our method improves robust accuracy by up to 40%, with a minimal impact on clean task accuracy.

翻译：尽管分布式机器学习在学术界受到广泛关注，但先前的研究仅独立考察了训练阶段或推理阶段的分布式机器学习实例。目前尚无研究探讨将学习过程与推理过程同时分布式化所产生的综合鲁棒性。本研究首次探索了在训练数据、架构、调度器、优化器及其他模型参数上完全异构的分布式机器学习模型的鲁棒性。通过理论分析及基于CIFAR10与FashionMNIST的广泛实验验证，我们证明此类经过合理分布式部署的机器学习实例能在准确率与鲁棒性的权衡中实现全面提升，有效抵御最先进的基于迁移的攻击——这是当前集成学习或联邦学习实例所无法实现的。例如，在CIFAR10数据集上针对当前最强大的迁移攻击之一Common Weakness攻击的实验表明，本方法可将鲁棒准确率提升高达40%，同时对原始任务准确率的影响微乎其微。