Federated Learning (FL) is a setting for training machine learning models in distributed environments where the clients do not share their raw data but instead send model updates to a server. However, model updates can be subject to attacks and leak private information. Differential Privacy (DP) is a leading mitigation strategy which involves adding noise to clipped model updates, trading off performance for strong theoretical privacy guarantees. Previous work has shown that the threat model of DP is conservative and that the obtained guarantees may be vacuous or may overestimate information leakage in practice. In this paper, we aim to achieve a tighter measurement of the model exposure by considering a realistic threat model. We propose a novel method, CANIFE, that uses canaries - carefully crafted samples by a strong adversary to evaluate the empirical privacy of a training round. We apply this attack to vision models trained on CIFAR-10 and CelebA and to language models trained on Sent140 and Shakespeare. In particular, in realistic FL scenarios, we demonstrate that the empirical per-round epsilon obtained with CANIFE is 4-5x lower than the theoretical bound.

翻译：联邦学习（FL）是一种分布式环境中训练机器学习模型的范式，其中客户端不共享原始数据，而是将模型更新发送至服务器。然而，模型更新可能遭受攻击并泄露隐私信息。差分隐私（DP）作为主要缓解策略，通过向裁剪后的模型更新添加噪声，以牺牲性能为代价换取严格的理论隐私保障。已有研究表明，DP的威胁模型较为保守，其获得的保障可能无效或在实际应用中高估信息泄露程度。本文旨在通过考虑现实的威胁模型，实现对模型暴露程度的更精确测量。我们提出了一种新颖方法CANIFE，该方法利用金丝雀样本——由强敌手精心构造的样本——来评估训练轮次的经验隐私水平。我们将该攻击应用于在CIFAR-10和CelebA上训练的视觉模型，以及在Sent140和莎士比亚数据集上训练的语言模型。特别地，在现实的联邦学习场景中，我们证明CANIFE获得的每轮经验epsilon值比理论边界低4至5倍。