We show the effectiveness of automatic differentiation in efficiently and correctly computing and controlling the spectrum of implicitly linear operators, a rich family of layer types including all standard convolutional and dense layers. We provide the first clipping method which is correct for general convolution layers, and illuminate the representational limitation that caused correctness issues in prior work. We study the effect of the batch normalization layers when concatenated with convolutional layers and show how our clipping method can be applied to their composition. By comparing the accuracy and performance of our algorithms to the state-of-the-art methods, using various experiments, we show they are more precise and efficient and lead to better generalization and adversarial robustness. We provide the code for using our methods at https://github.com/Ali-E/FastClip.

翻译：我们展示了自动微分在高效且精确计算与控制隐式线性算子谱方面的有效性。隐式线性算子是一类丰富的层类型，包含所有标准卷积层和全连接层。我们提出了首个适用于通用卷积层的正确裁剪方法，并揭示了先前工作中因表征限制导致的正确性问题。通过研究批归一化层与卷积层级联时的特性，我们展示了如何将所提裁剪方法应用于其组合结构。通过多组实验对比我们算法与现有最优方法的精度与性能，证明本方法具有更高的精确度与效率，并能带来更好的泛化能力与对抗鲁棒性。相关代码已开源在 https://github.com/Ali-E/FastClip。