Recent developments in deep learning have led to great success in various natural language processing (NLP) tasks. However, these applications may involve data that contain sensitive information. Therefore, how to achieve good performance while also protecting the privacy of sensitive data is a crucial challenge in NLP. To preserve privacy, Differential Privacy (DP), which can prevent reconstruction attacks and protect against potential side knowledge, is becoming a de facto technique for private data analysis. In recent years, NLP in DP models (DP-NLP) has been studied from different perspectives, which deserves a comprehensive review. In this paper, we provide the first systematic review of recent advances in DP deep learning models in NLP. In particular, we first discuss some differences and additional challenges of DP-NLP compared with the standard DP deep learning. Then, we investigate some existing work on DP-NLP and present its recent developments from three aspects: gradient perturbation based methods, embedding vector perturbation based methods, and ensemble model based methods. We also discuss some challenges and future directions.

翻译：深度学习的最新进展在各类自然语言处理任务中取得了巨大成功。然而，这些应用可能涉及包含敏感信息的数据。因此，如何在保持良好性能的同时保护敏感数据的隐私，是自然语言处理领域的一项关键挑战。为保护隐私，差分隐私作为一种能够防止重构攻击并抵御潜在侧面知识的技术，正成为私有数据分析的事实标准。近年来，基于差分隐私模型的自然语言处理研究从不同角度展开，亟需系统综述。本文首次系统梳理了自然语言处理中差分隐私深度学习模型的最新进展。我们首先讨论了差分隐私自然语言处理相较于标准差分隐私深度学习的差异与额外挑战；接着从梯度扰动方法、嵌入向量扰动方法和集成模型方法三个层面，考察了现有相关工作并呈现其最新发展；最后探讨了相关挑战与未来方向。