In Federated Learning (FL), accessing private client data incurs communication and privacy costs. As a result, FL deployments commonly prefinetune pretrained foundation models on a (large, possibly public) dataset that is held by the central server; they then FL-finetune the model on a private, federated dataset held by clients. Evaluating prefinetuning dataset quality reliably and privately is therefore of high importance. To this end, we propose FreD (Federated Private Fr\'echet Distance) -- a privately computed distance between a prefinetuning dataset and federated datasets. Intuitively, it privately computes and compares a Fr\'echet distance between embeddings generated by a large language model on both the central (public) dataset and the federated private client data. To make this computation privacy-preserving, we use distributed, differentially-private mean and covariance estimators. We show empirically that FreD accurately predicts the best prefinetuning dataset at minimal privacy cost. Altogether, using FreD we demonstrate a proof-of-concept for a new approach in private FL training: (1) customize a prefinetuning dataset to better match user data (2) prefinetune (3) perform FL-finetuning.

翻译：摘要：在联邦学习（FL）中，访问私有客户端数据会带来通信和隐私成本。因此，联邦学习部署通常先在中央服务器持有的（大型、可能公开的）数据集上对预训练基础模型进行预微调，随后在客户端持有的私有联邦数据集上进行联邦微调。因此，可靠且隐私地评估预微调数据集的质量至关重要。为此，我们提出FreD（联邦私有弗雷歇距离）——一种在预微调数据集与联邦数据集之间进行隐私计算的度量指标。直观上，该方法通过隐私计算方式，在大语言模型生成的公共数据集与联邦私有客户端数据的嵌入表示之间，计算并比较弗雷歇距离。为实现隐私保护计算，我们采用分布式差分隐私均值与协方差估计器。实验证明，FreD能以最小隐私代价准确预测最佳预微调数据集。综上，我们通过FreD为私有联邦训练开创了新方法概念验证：（1）自定义预微调数据集以更好匹配用户数据（2）进行预微调（3）执行联邦微调。