Recurrent neural networks are extremely powerful yet hard to train. One of their issues is the vanishing gradient problem, whereby propagation of training signals may be exponentially attenuated, freezing training. Use of orthogonal or unitary matrices, whose powers neither explode nor decay, has been proposed to mitigate this issue, but their computational expense has hindered their use. Here we show that in the specific case of convolutional RNNs, we can define a convolutional exponential and that this operation transforms antisymmetric or anti-Hermitian convolution kernels into orthogonal or unitary convolution kernels. We explicitly derive FFT-based algorithms to compute the kernels and their derivatives. The computational complexity of parametrizing this subspace of orthogonal transformations is thus the same as the networks' iteration.

翻译：循环神经网络虽极为强大却难以训练。其中一个问题是梯度消失现象，即训练信号的传播可能呈指数级衰减，导致训练冻结。采用幂次既不会爆炸也不会衰减的正交矩阵或酉矩阵来缓解该问题的方法已被提出，但其计算成本阻碍了实际应用。本文证明，在卷积循环神经网络的特例中，我们可以定义卷积指数函数，该运算能将反对称或反厄米卷积核转化为正交或酉卷积核。我们明确推导了基于快速傅里叶变换的算法来计算这些卷积核及其导数。因此，参数化这一正交变换子空间的计算复杂度与网络迭代的计算复杂度相同。