State-of-the-art approaches for training Differentially Private (DP) Deep Neural Networks (DNN) faces difficulties to estimate tight bounds on the sensitivity of the network's layers, and instead rely on a process of per-sample gradient clipping. This clipping process not only biases the direction of gradients but also proves costly both in memory consumption and in computation. To provide sensitivity bounds and bypass the drawbacks of the clipping process, our theoretical analysis of Lipschitz constrained networks reveals an unexplored link between the Lipschitz constant with respect to their input and the one with respect to their parameters. By bounding the Lipschitz constant of each layer with respect to its parameters we guarantee DP training of these networks. This analysis not only allows the computation of the aforementioned sensitivities at scale but also provides leads on to how maximize the gradient-to-noise ratio for fixed privacy guarantees. To facilitate the application of Lipschitz networks and foster robust and certifiable learning under privacy guarantees, we provide a Python package that implements building blocks allowing the construction and private training of such networks.

翻译：当前训练差分隐私深度神经网络的最先进方法在估计网络层灵敏度紧界方面存在困难，转而依赖逐样本梯度裁剪过程。该裁剪过程不仅会偏置梯度方向，还会在内存消耗和计算方面产生高昂成本。为提供灵敏度界并规避裁剪过程的缺陷，我们对利普希茨约束网络的理论分析揭示了关于输入利普希茨常数与参数利普希茨常数之间尚未被探索的联系。通过约束每层关于参数的利普希茨常数，我们确保了这些网络的差分隐私训练。该分析不仅能实现上述灵敏度的大规模计算，还能为在固定隐私保证下最大化梯度噪声比提供指导。为促进利普希茨网络的应用并在隐私保证下实现稳健且可认证的学习，我们提供了一个Python软件包，其中实现了用于构建和私有训练此类网络的构建模块。