Parallelization techniques have become ubiquitous for accelerating inference and training of deep neural networks. Despite this, several operations are still performed in a sequential manner. For instance, the forward and backward passes are executed layer-by-layer, and the output of diffusion models is produced by applying a sequence of denoising steps. This sequential approach results in a computational cost proportional to the number of steps involved, presenting a potential bottleneck as the number of steps increases. In this work, we introduce DeepPCR, a novel algorithm which parallelizes typically sequential operations in order to speed up inference and training of neural networks. DeepPCR is based on interpreting a sequence of $L$ steps as the solution of a specific system of equations, which we recover using the Parallel Cyclic Reduction algorithm. This reduces the complexity of computing the sequential operations from $\mathcal{O}(L)$ to $\mathcal{O}(\log_2L)$, thus yielding a speedup for large $L$. To verify the theoretical lower complexity of the algorithm, and to identify regimes for speedup, we test the effectiveness of DeepPCR in parallelizing the forward and backward pass in multi-layer perceptrons, and reach speedups of up to $30\times$ for the forward and $200\times$ for the backward pass. We additionally showcase the flexibility of DeepPCR by parallelizing training of ResNets with as many as 1024 layers, and generation in diffusion models, enabling up to $7\times$ faster training and $11\times$ faster generation, respectively, when compared to the sequential approach.

翻译：并行化技术已成为加速深度神经网络推理与训练的通用方法。尽管如此，若干操作仍以顺序方式执行。例如，前向传播与反向传播逐层执行，扩散模型的输出通过连续去噪步骤生成。这种顺序方法导致计算成本与操作步数成正比，当步数增加时可能形成性能瓶颈。本文提出DeepPCR算法，这是一种新型算法，通过并行化典型顺序操作来加速神经网络的推理与训练。DeepPCR的核心思想是将$L$步序列操作视为特定方程组的解，并采用并行循环约简算法求解。该算法将顺序操作的计算复杂度从$\mathcal{O}(L)$降至$\mathcal{O}(\log_2L)$，从而在$L$较大时实现加速。为验证算法的理论低复杂度特性并识别加速场景，我们在多层感知机的前向传播与反向传播中测试了DeepPCR的并行化效果，分别实现了前向传播最高30倍、反向传播最高200倍的加速。此外，我们展示了DeepPCR的灵活性：通过并行化训练含1024层的残差网络（ResNets）以及扩散模型的生成过程，相比顺序方法，训练速度最高提升7倍，生成速度最高提升11倍。