Due to constraints on privacy, cost, and latency, on-premise deployment of small models is increasingly common. However, most practical pipelines stop at supervised fine-tuning (SFT) and fail to reach the reinforcement learning (RL) alignment stage. The main reason is that RL alignment typically requires either expensive human preference annotation or heavy reliance on high-quality reward models with large-scale API usage and ongoing engineering maintenance, both of which are ill-suited to on-premise settings. To bridge this gap, we propose a positive-unlabeled (PU) RL distillation method for on-premise small-model deployment. Without human-labeled preferences or a reward model, our method distills the teacher's preference-optimization capability from black-box generations into a locally trainable student. For each prompt, we query the teacher once to obtain an anchor response, locally sample multiple student candidates, and perform anchor-conditioned self-ranking to induce pairwise or listwise preferences, enabling a fully local training loop via direct preference optimization or group relative policy optimization. Theoretical analysis justifies that the induced preference signal by our method is order-consistent and concentrates on near-optimal candidates, supporting its stability for preference optimization. Experiments demonstrate that our method achieves consistently strong performance under a low-cost setting.

翻译：由于隐私、成本与延迟方面的限制，小模型的本地部署日益普遍。然而，大多数实际应用流程仅停留在监督微调阶段，未能进入强化学习对齐环节。其主要原因在于，强化学习对齐通常需要昂贵的人工偏好标注，或严重依赖高质量奖励模型并伴随大规模API调用与持续的工程维护，这两者均不适用于本地部署场景。为弥合这一差距，我们提出了一种面向本地部署小模型的积极-无标签强化学习蒸馏方法。该方法无需人工标注的偏好数据或奖励模型，而是将教师模型基于偏好优化的能力从其黑箱生成结果中蒸馏至可本地训练的学生模型。针对每个提示，我们向教师模型进行一次查询以获得锚定响应，在本地采样多个学生候选响应，并通过锚定条件自排序来推导成对或列表形式的偏好，从而通过直接偏好优化或分组相对策略优化实现完全本地的训练循环。理论分析证明，本方法推导出的偏好信号具有顺序一致性，并集中于接近最优的候选响应，这支撑了其在偏好优化过程中的稳定性。实验结果表明，本方法在低成本设定下能持续取得强劲的性能表现。