Retrieval-based language models (LMs) have demonstrated improved interpretability, factuality, and adaptability compared to their parametric counterparts, by incorporating retrieved text from external datastores. While it is well known that parametric models are prone to leaking private data, it remains unclear how the addition of a retrieval datastore impacts model privacy. In this work, we present the first study of privacy risks in retrieval-based LMs, particularly $k$NN-LMs. Our goal is to explore the optimal design and training procedure in domains where privacy is of concern, aiming to strike a balance between utility and privacy. Crucially, we find that $k$NN-LMs are more susceptible to leaking private information from their private datastore than parametric models. We further explore mitigations of privacy risks. When privacy information is targeted and readily detected in the text, we find that a simple sanitization step would completely eliminate the risks, while decoupling query and key encoders achieves an even better utility-privacy trade-off. Otherwise, we consider strategies of mixing public and private data in both datastore and encoder training. While these methods offer modest improvements, they leave considerable room for future work. Together, our findings provide insights for practitioners to better understand and mitigate privacy risks in retrieval-based LMs. Our code is available at: https://github.com/Princeton-SysML/kNNLM_privacy .

翻译：基于检索的语言模型通过整合外部数据存储中的检索文本，相较于纯参数化模型，在可解释性、事实准确性和适应性方面均有提升。尽管参数化模型容易泄露隐私数据已是共识，但检索数据存储的加入如何影响模型隐私仍不明确。本文首次对基于检索的语言模型（特别是$k$近邻语言模型，$k$NN-LMs）的隐私风险进行研究。我们旨在探索隐私敏感领域中的最优设计与训练流程，以平衡效用与隐私。关键发现是：相较于参数化模型，$k$NN-LMs更易从其私有数据存储中泄露隐私信息。我们进一步探索了隐私风险的缓解措施。当隐私信息具有明确指向性且易于在文本中检测时，简单的数据清洗步骤即可完全消除风险，而解耦查询编码器与关键编码器的方法则能实现更优的效用-隐私权衡。在其他情况下，我们考虑了在数据存储和编码器训练中混合公开与私有数据的策略。尽管这些方法带来了一定改进，但仍为未来研究留下了充足空间。综合而言，我们的研究为实践者理解并缓解基于检索的语言模型中的隐私风险提供了洞见。我们的代码开源地址为：https://github.com/Princeton-SysML/kNNLM_privacy。