We present a general variational framework for the training of freeform nonlinearities in layered computational architectures subject to some slope constraints. The regularization that we add to the traditional training loss penalizes the second-order total variation of each trainable activation. The slope constraints allow us to impose properties such as 1-Lipschitz stability, firm non-expansiveness, and monotonicity/invertibility. These properties are crucial to ensure the proper functioning of certain classes of signal-processing algorithms (e.g., plug-and-play schemes, unrolled proximal gradient, invertible flows). We prove that the global optimum of the stated constrained-optimization problem is achieved with nonlinearities that are adaptive nonuniform linear splines. We then show how to solve the resulting function-optimization problem numerically by representing the nonlinearities in a suitable (nonuniform) B-spline basis. Finally, we illustrate the use of our framework with the data-driven design of (weakly) convex regularizers for the denoising of images and the resolution of inverse problems.

翻译：我们提出了一种通用的变分框架，用于训练受斜率约束的分层计算架构中的自由形式非线性。我们在传统训练损失基础上添加的正则化项惩罚每个可训练激活函数的二阶全变差。斜率约束允许我们施加诸如1-李普希茨稳定性、严格非扩张性以及单调性/可逆性等特性。这些特性对于确保特定类别信号处理算法（例如即插即用方案、展开近端梯度法、可逆流）的正常运行至关重要。我们证明了所述约束优化问题的全局最优解可通过自适应非均匀线性样条非线性实现。随后，我们展示了如何通过在合适的（非均匀）B样条基中表示非线性函数，对所得函数优化问题进行数值求解。最后，我们通过数据驱动的（弱）凸正则化器设计在图像去噪和逆问题求解中的应用，展示了该框架的实际效用。