End-to-end backpropagation requires storing activations throughout all layers, creating memory bottlenecks that limit model scalability. Existing block-wise training methods offer means to alleviate this problem, but they rely on ad-hoc local objectives and remain largely unexplored beyond classification tasks. We propose $\textit{DiffusionBlocks}$, a principled framework for transforming transformer-based networks into genuinely independent trainable blocks that maintain competitive performance with end-to-end training. Our key insight leverages the fact that residual connections naturally correspond to updates in a dynamical system. With minimal modifications to this system, we can convert the updates to those of a denoising process, where each block can be learned independently by leveraging the score matching objective. This independence enables training with gradients for only one block at a time, thereby reducing memory requirements in proportion to the number of blocks. Our experiments on a range of transformer architectures (vision, diffusion, autoregressive, recurrent-depth, and masked diffusion) demonstrate that DiffusionBlocks training matches the performance of end-to-end training while enabling scalable block-wise training on practical tasks beyond small-scale classification. DiffusionBlocks provides a theoretically grounded approach that successfully scales to modern generative tasks across diverse architectures. Code is available at https://github.com/SakanaAI/DiffusionBlocks .

翻译：端到端的反向传播需要在所有层中存储激活值，这造成了限制模型可扩展性的内存瓶颈。现有的逐块训练方法提供了缓解该问题的途径，但其依赖于临时性的局部目标，并且除分类任务外仍基本未被探索。我们提出$\textit{DiffusionBlocks}$，这是一个将基于Transformer的网络转化为真正独立可训练块（在保持与端到端训练竞争性性能的同时）的基于原则的框架。我们的关键见解利用了一个事实：残差连接自然对应于动力系统中的更新。通过对该系统进行最小修改，我们可以将这些更新转化为去噪过程中的更新，其中每个块可以通过利用分数匹配目标独立学习。这种独立性使得一次只能针对一个块进行梯度训练，从而将内存需求按块的数量成比例减少。我们在多种Transformer架构（视觉、扩散、自回归、循环深度和掩码扩散）上的实验表明，DiffusionBlocks训练在匹配端到端训练性能的同时，能够在超出小规模分类的实际任务上实现可扩展的逐块训练。DiffusionBlocks提供了一种有理论依据的方法，成功扩展到跨多种架构的现代生成任务。代码可在https://github.com/SakanaAI/DiffusionBlocks获取。