The adoption of large cloud-based models for inference has been hampered by concerns about the privacy leakage of end-user data. One method to mitigate this leakage is to add local differentially private noise to queries before sending them to the cloud, but this degrades utility as a side effect. Our key insight is that knowledge available in the noisy labels returned from performing inference on noisy inputs can be aggregated and used to recover the correct labels. We implement this insight in LDPKiT, which stands for Local Differentially-Private and Utility-Preserving Inference via Knowledge Transfer. LDPKiT uses the noisy labels returned from querying a set of noised inputs to train a local model (noise^2), which is then used to perform inference on the original set of inputs. Our experiments on CIFAR-10, Fashion-MNIST, SVHN, and CARER NLP datasets demonstrate that LDPKiT can improve utility without compromising privacy. For instance, on CIFAR-10, compared to a standard $\epsilon$-LDP scheme with $\epsilon=15$, which provides a weak privacy guarantee, LDPKiT can achieve nearly the same accuracy (within 1% drop) with $\epsilon=7$, offering an enhanced privacy guarantee. Moreover, the benefits of using LDPKiT increase at higher, more privacy-protective noise levels. For Fashion-MNIST and CARER, LDPKiT's accuracy on the sensitive dataset with $\epsilon=7$ not only exceeds the average accuracy of the standard $\epsilon$-LDP scheme with $\epsilon=7$ by roughly 20% and 9% but also outperforms the standard $\epsilon$-LDP scheme with $\epsilon=15$, a scenario with less noise and minimal privacy protection. We also perform Zest distance measurements to demonstrate that the type of distillation performed by LDPKiT is different from a model extraction attack.

翻译：采用基于云端的大型模型进行推理一直受到对终端用户数据隐私泄露担忧的阻碍。缓解这种泄露的一种方法是在将查询发送到云端之前，向其中添加本地差分隐私噪声，但这会降低效用作为副作用。我们的核心见解是，通过对噪声输入执行推理所返回的噪声标签中可用的知识可以被聚合并用于恢复正确的标签。我们在LDPKiT中实现了这一见解，LDPKiT代表通过知识转移实现本地差分隐私且保持效用的推理。LDPKiT使用查询一组噪声输入所返回的噪声标签来训练一个本地模型（噪声平方），然后使用该模型对原始输入集执行推理。我们在CIFAR-10、Fashion-MNIST、SVHN和CARER NLP数据集上的实验表明，LDPKiT可以在不损害隐私的情况下提高效用。例如，在CIFAR-10上，与提供较弱隐私保证的标准$\epsilon$-LDP方案（$\epsilon=15$）相比，LDPKiT可以在$\epsilon=7$（提供更强的隐私保证）下实现几乎相同的准确率（下降在1%以内）。此外，在更高、更具隐私保护性的噪声水平下，使用LDPKiT的优势会增加。对于Fashion-MNIST和CARER，LDPKiT在敏感数据集上使用$\epsilon=7$的准确率，不仅比标准$\epsilon$-LDP方案在$\epsilon=7$下的平均准确率高出约20%和9%，而且优于噪声较少、隐私保护最少的场景下标准$\epsilon$-LDP方案在$\epsilon=15$下的准确率。我们还进行了Zest距离测量，以证明LDPKiT执行的蒸馏类型不同于模型提取攻击。