We introduce UniCon, a novel architecture designed to enhance control and efficiency in training adapters for large-scale diffusion models. Unlike existing methods that rely on bidirectional interaction between the diffusion model and control adapter, UniCon implements a unidirectional flow from the diffusion network to the adapter, allowing the adapter alone to generate the final output. UniCon reduces computational demands by eliminating the need for the diffusion model to compute and store gradients during adapter training. Our results indicate that UniCon reduces GPU memory usage by one-third and increases training speed by 2.3 times, while maintaining the same adapter parameter size. Additionally, without requiring extra computational resources, UniCon enables the training of adapters with double the parameter volume of existing ControlNets. In a series of image conditional generation tasks, UniCon has demonstrated precise responsiveness to control inputs and exceptional generation capabilities.

翻译：本文提出UniCon，一种旨在增强大规模扩散模型适配器训练控制能力与效率的新型架构。与现有方法依赖扩散模型与控制适配器之间的双向交互不同，UniCon实现了从扩散网络到适配器的单向信息流，使得仅需适配器即可生成最终输出。UniCon通过消除适配器训练期间扩散模型计算和存储梯度的需求，显著降低了计算负担。实验结果表明，在保持相同适配器参数量的前提下，UniCon将GPU内存占用减少三分之一，并将训练速度提升2.3倍。此外，在无需额外计算资源的情况下，UniCon能够训练参数量达到现有ControlNet两倍的适配器。在一系列图像条件生成任务中，UniCon展现出对控制信号的精确响应能力与卓越的生成性能。