Recent studies improve on-device language model (LM) inference through end-cloud collaboration, where the end device retrieves useful information from cloud databases to enhance local processing, known as Retrieval-Augmented Generation (RAG). Typically, to retrieve information from the cloud while safeguarding privacy, the end device transforms original data into embeddings with a local embedding model. However, the recently emerging Embedding Inversion Attacks (EIAs) can still recover the original data from text embeddings (e.g., training a recovery model to map embeddings back to original texts), posing a significant threat to user privacy. To address this risk, we propose EntroGuard, an entropy-driven perturbation-based embedding privacy protection method, which can protect the privacy of text embeddings while maintaining retrieval accuracy during the end-cloud collaboration. Specifically, to defeat various EIAs, we perturb the embeddings to increase the entropy of the recovered text in the common structure of recovery models, thus steering the embeddings toward meaningless texts rather than original sensitive texts during the recovery process. To maintain retrieval performance in the cloud, we constrain the perturbations within a bound, applying the strategy of reducing them where redundant and increasing them where sparse. Moreover, EntroGuard can be directly integrated into end devices without requiring any modifications to the embedding model. Extensive experimental results demonstrate that EntroGuard can reduce the risk of privacy leakage by up to 8 times at most with negligible loss of retrieval performance compared to existing privacy-preserving methods.

翻译：近期研究通过端云协作改进了设备端语言模型（LM）的推理能力，其中终端设备从云端数据库检索有用信息以增强本地处理，即检索增强生成（RAG）。通常，为了在从云端检索信息的同时保护隐私，终端设备会使用本地嵌入模型将原始数据转换为嵌入向量。然而，新近出现的嵌入反演攻击（EIAs）仍能从文本嵌入中恢复原始数据（例如，通过训练恢复模型将嵌入映射回原始文本），这对用户隐私构成了重大威胁。为应对此风险，我们提出了EntroGuard——一种基于熵驱动的扰动的嵌入隐私保护方法，该方法能够在端云协作过程中保护文本嵌入的隐私，同时保持检索准确性。具体而言，为抵御各类EIA，我们对嵌入向量施加扰动，以增加恢复模型常见结构中恢复文本的熵，从而在恢复过程中引导嵌入向量趋向无意义的文本而非原始的敏感文本。为维持云端检索性能，我们将扰动限制在一定范围内，并采用"冗余处减少、稀疏处增加"的策略。此外，EntroGuard可直接集成到终端设备中，无需对嵌入模型进行任何修改。大量实验结果表明，与现有隐私保护方法相比，EntroGuard在检索性能损失可忽略不计的情况下，最多可将隐私泄露风险降低达8倍。