Differentially private stochastic gradient descent privatizes model training by injecting noise into each iteration, where the noise magnitude increases with the number of model parameters. Recent works suggest that we can reduce the noise by leveraging public data for private machine learning, by projecting gradients onto a subspace prescribed by the public data. However, given a choice of public datasets, it is not a priori clear which one may be most appropriate for the private task. We give an algorithm for selecting a public dataset by measuring a low-dimensional subspace distance between gradients of the public and private examples. We provide theoretical analysis demonstrating that the excess risk scales with this subspace distance. This distance is easy to compute and robust to modifications in the setting. Empirical evaluation shows that trained model accuracy is monotone in this distance.

翻译：差分隐私随机梯度下降通过向每次迭代注入噪声来对模型训练进行隐私保护，其中噪声幅度随模型参数数量增加而增大。近期研究表明，通过将梯度投影到公共数据规定的子空间上，可借助公共数据降低私有机器学习中的噪声。然而，面对多个候选公共数据集时，事先难以明确哪个最适合私有任务。我们提出了一种通过测量公共数据与私有数据梯度间的低维子空间距离来选择公共数据集的算法。理论分析表明，过量风险与该子空间距离成正比。该距离计算简便，且对设置修改具有鲁棒性。实验评估显示，训练后的模型准确率与该距离呈单调关系。