We study the inherent trade-offs in minimizing privacy risks and maximizing utility, while maintaining high computational efficiency, when fine-tuning large language models (LLMs). A number of recent works in privacy research have attempted to mitigate privacy risks posed by memorizing fine-tuning data by using differentially private training methods (e.g., DP), albeit at a significantly higher computational cost (inefficiency). In parallel, several works in systems research have focussed on developing (parameter) efficient fine-tuning methods (e.g., LoRA), but few works, if any, investigated whether such efficient methods enhance or diminish privacy risks. In this paper, we investigate this gap and arrive at a surprising conclusion: efficient fine-tuning methods like LoRA mitigate privacy risks similar to private fine-tuning methods like DP. Our empirical finding directly contradicts prevailing wisdom that privacy and efficiency objectives are at odds during fine-tuning. Our finding is established by (a) carefully defining measures of privacy and utility that distinguish between memorizing sensitive and non-sensitive tokens in training and test datasets used in fine-tuning and (b) extensive evaluations using multiple open-source language models from Pythia, Gemma, Llama, and Qwen families and different domain-specific datasets.

翻译：本研究探讨在微调大语言模型（LLMs）时，如何权衡最小化隐私风险、最大化模型效用，并保持高计算效率。隐私研究领域近期多项工作尝试通过差分隐私训练方法（如DP）来缓解因记忆微调数据而产生的隐私风险，但这通常伴随着显著增加的计算成本（低效率）。与此同时，系统研究领域的若干工作专注于开发（参数）高效的微调方法（如LoRA），但很少有研究探讨此类高效方法会增强还是削弱隐私风险。本文针对这一空白展开研究，并得出一个令人惊讶的结论：LoRA等高效微调方法能够缓解隐私风险，其效果与DP等隐私保护微调方法类似。我们的实证发现直接挑战了当前普遍认为的“微调过程中隐私目标与效率目标相互冲突”的观点。这一结论的建立基于：（a）精确定义隐私与效用的度量指标，以区分微调所用训练集和测试集中敏感与非敏感标记的记忆情况；（b）使用来自Pythia、Gemma、Llama和Qwen系列的多个开源语言模型及不同领域专用数据集进行的广泛评估。