This paper derives a complete set of quadratic constraints (QCs) for the repeated ReLU. The complete set of QCs is described by a collection of matrix copositivity conditions. We also show that only two functions satisfy all QCs in our complete set: the repeated ReLU and flipped ReLU. Thus our complete set of QCs bounds the repeated ReLU as tight as possible up to the sign invariance inherent in quadratic forms. We derive a similar complete set of incremental QCs for repeated ReLU, which can potentially lead to less conservative Lipschitz bounds for ReLU networks than the standard LipSDP approach. The basic constructions are also used to derive the complete sets of QCs for other piecewise linear activation functions such as leaky ReLU, MaxMin, and HouseHolder. Finally, we illustrate the use of the complete set of QCs to assess stability and performance for recurrent neural networks with ReLU activation functions. We rely on a standard copositivity relaxation to formulate the stability/performance condition as a semidefinite program. Simple examples are provided to illustrate that the complete sets of QCs and incremental QCs can yield less conservative bounds than existing sets.

翻译：本文推导了重复ReLU的完整二次约束集。该完整二次约束集由一组矩阵余正性条件描述。我们还证明了仅有两个函数满足完整集中的所有二次约束：重复ReLU与翻转ReLU。因此，我们的完整二次约束集在二次形式固有的符号不变性范围内，以尽可能紧密的方式界定了重复ReLU。我们进一步推导了重复ReLU的完整增量二次约束集，相较于标准LipSDP方法，该集合可能为ReLU网络带来更保守的Lipschitz界。基本构造方法同样适用于推导其他分段线性激活函数（如leaky ReLU、MaxMin和HouseHolder）的完整二次约束集。最后，我们通过示例说明如何利用完整二次约束集评估具有ReLU激活函数的循环神经网络的稳定性和性能。我们采用标准余正性松弛方法，将稳定性/性能条件表述为半定规划问题。提供的简单算例表明，完整二次约束集与增量二次约束集能够产生比现有约束集更保守的界。