The nonparametric variational information bottleneck (NVIB) provides the foundation for nonparametric variational differential privacy (NVDP), a framework for building privacy-preserving language models. However, the learned latent representations can drift into regions with high information content, leading to poor privacy guarantees, but also low utility due to numerical instability during training. In this work, we introduce a principled parameter clipping strategy to directly address this issue. Our method is mathematically derived from the objective of minimizing the Rényi Divergence (RD) upper bound, yielding specific, theoretically grounded constraints on the posterior mean, variance, and mixture weight parameters. We apply our technique to an NVIB based model and empirically compare it against an unconstrained baseline. Our findings demonstrate that the clipped model consistently achieves tighter RD bounds, implying stronger privacy, while simultaneously attaining higher performance on several downstream tasks. This work presents a simple yet effective method for improving the privacy-utility trade-off in variational models, making them more robust and practical.

翻译：非参数变分信息瓶颈（NVIB）为非参数变分差分隐私（NVDP）奠定了基础，后者是一种构建隐私保护语言模型的框架。然而，学习得到的潜在表示可能会漂移到信息量较高的区域，导致隐私保障性差，同时由于训练过程中的数值不稳定性而导致效用低下。在本文中，我们引入了一种原理性的参数裁剪策略来直接解决这一问题。我们的方法在数学上源于最小化瑞利散度上界的目标，从而对后验均值、方差和混合权重参数施加了具体且具有理论依据的约束。我们将该技术应用于基于NVIB的模型，并与无约束基线进行实证比较。我们的研究结果表明，裁剪后的模型始终能获得更紧的瑞利散度上界，这意味着更强的隐私性，同时在多个下游任务中获得更高的性能。这项工作为改进变分模型中的隐私-效用权衡提供了一种简单而有效的方法，使其更具鲁棒性和实用性。