With the advancement of language models (LMs), their exposure to private data is increasingly inevitable, and their deployment (especially for smaller ones) on personal devices, such as PCs and smartphones, has become a prevailing trend. In contexts laden with user information, enabling models to both safeguard user privacy and execute commands efficiently emerges as an essential research imperative. In this paper, we propose CoGenesis, a collaborative generation framework integrating large (hosted on cloud infrastructure) and small models (deployed on local devices) to address privacy concerns logically. Initially, we design a pipeline to create personalized writing instruction datasets enriched with extensive context details as the testbed of this research issue. Subsequently, we introduce two variants of CoGenesis based on sketch and logits respectively. Our experimental findings, based on our synthesized dataset and two additional open-source datasets, indicate that: 1) Large-scale models perform well when provided with user context but struggle in the absence of such context. 2) While specialized smaller models fine-tuned on the synthetic dataset show promise, they still lag behind their larger counterparts. 3) Our CoGenesis framework, utilizing mixed-scale models, showcases competitive performance, providing a feasible solution to privacy issues.

翻译：随着语言模型的发展，其接触私有数据的情况日益不可避免，而模型（尤其是小规模模型）在个人设备（如个人电脑和智能手机）上的部署已成为主流趋势。在充满用户信息的情境中，使模型既能保护用户隐私又能高效执行命令成为一项重要的研究任务。本文提出CoGenesis，一种集成大模型（部署于云基础设施）与小模型（部署于本地设备）的协同生成框架，旨在从逻辑层面解决隐私问题。首先，我们设计了一条流程，用于创建包含丰富上下文细节的个性化写作指令数据集，作为该研究问题的测试平台。随后，我们分别介绍了基于草图（sketch）和基于logits的两种CoGenesis变体。基于我们合成的数据集及两个额外开源数据集的实验结果表明：1）大规模模型在提供用户上下文时表现良好，但在缺乏此类上下文时则表现不佳；2）在合成数据集上微调的专用小模型虽展现出潜力，但仍落后于其大规模对应模型；3）我们的CoGenesis框架通过使用混合规模模型展现出具有竞争力的性能，为隐私问题提供了可行的解决方案。