Preserving privacy in sensitive data while pretraining large language models on small, domain-specific corpora presents a significant challenge. In this work, we take an exploratory step toward privacy-preserving continual pretraining by proposing an entity-based framework that synthesizes encrypted training data to protect personally identifiable information (PII). Our approach constructs a weighted entity graph to guide data synthesis and applies deterministic encryption to PII entities, enabling LLMs to encode new knowledge through continual pretraining while granting authorized access to sensitive data through decryption keys. Our results on limited-scale datasets demonstrate that our pretrained models outperform base models and ensure PII security, while exhibiting a modest performance gap compared to models trained on unencrypted synthetic data. We further show that increasing the number of entities and leveraging graph-based synthesis improves model performance, and that encrypted models retain instruction-following capabilities with long retrieved contexts. We discuss the security implications and limitations of deterministic encryption, positioning this work as an initial investigation into the design space of encrypted data pretraining for privacy-preserving LLMs. Our code is available at https://github.com/DataArcTech/SoE.

翻译：在小型领域特定语料库上预训练大语言模型时保护敏感数据隐私是一项重大挑战。本研究通过提出一种基于实体的框架，合成加密训练数据以保护个人可识别信息，向隐私保护的持续预训练迈出了探索性步伐。我们的方法构建加权实体图谱来指导数据合成，并对PII实体应用确定性加密，使大语言模型能够通过持续预训练编码新知识，同时通过解密密钥授予对敏感数据的授权访问。我们在有限规模数据集上的实验结果表明，预训练模型性能优于基础模型且能确保PII安全性，但与未加密合成数据训练的模型相比存在适度性能差距。我们进一步证明，增加实体数量与利用基于图谱的合成策略可提升模型性能，且加密模型在长检索上下文场景中仍保持指令跟随能力。我们探讨了确定性加密的安全影响与局限性，将本研究定位为隐私保护大语言模型加密数据预训练设计空间的初步探索。代码已开源：https://github.com/DataArcTech/SoE。