While open Large Language Models (LLMs) have made significant progress, they still fall short of matching the performance of their closed, proprietary counterparts, making the latter attractive even for the use on highly private data. Recently, various new methods have been proposed to adapt closed LLMs to private data without leaking private information to third parties and/or the LLM provider. In this work, we analyze the privacy protection and performance of the four most recent methods for private adaptation of closed LLMs. By examining their threat models and thoroughly comparing their performance under different privacy levels according to differential privacy (DP), various LLM architectures, and multiple datasets for classification and generation tasks, we find that: (1) all the methods leak query data, i.e., the (potentially sensitive) user data that is queried at inference time, to the LLM provider, (2) three out of four methods also leak large fractions of private training data to the LLM provider while the method that protects private data requires a local open LLM, (3) all the methods exhibit lower performance compared to three private gradient-based adaptation methods for local open LLMs, and (4) the private adaptation methods for closed LLMs incur higher monetary training and query costs than running the alternative methods on local open LLMs. This yields the conclusion that, to achieve truly privacy-preserving LLM adaptations that yield high performance and more privacy at lower costs, taking into account current methods and models, one should use open LLMs.

翻译：尽管开放大语言模型已取得显著进展，但其性能仍未能匹敌封闭的专有模型，这使得后者即使应用于高度私有数据也颇具吸引力。近期，学界提出了多种新方法，旨在将封闭大语言模型适配至私有数据，同时避免向第三方及大语言模型提供商泄露私有信息。本研究分析了四种最新的封闭大语言模型私有化适配方法的隐私保护能力与性能。通过审视其威胁模型，并依据差分隐私在不同隐私级别、多种大语言模型架构以及分类与生成任务的多类数据集下对其性能进行全面比较，我们发现：（1）所有方法均向大语言模型提供商泄露查询数据（即推理时查询的潜在敏感用户数据）；（2）四种方法中有三种还会向大语言模型提供商泄露大量私有训练数据，而能保护私有数据的方法需要依赖本地开放大语言模型；（3）与三种基于梯度的本地开放大语言模型私有化适配方法相比，所有方法的性能均较低；（4）封闭大语言模型的私有化适配方法在训练与查询成本上高于在本地开放大语言模型上运行替代方案。由此得出结论：基于当前方法与模型，若要实现真正保护隐私、性能优异、隐私性更强且成本更低的大语言模型适配，应当采用开放大语言模型。