ControlNets are widely used for adding spatial control to text-to-image diffusion models with different conditions, such as depth maps, scribbles/sketches, and human poses. However, when it comes to controllable video generation, ControlNets cannot be directly integrated into new backbones due to feature space mismatches, and training ControlNets for new backbones can be a significant burden for many users. Furthermore, applying ControlNets independently to different frames cannot effectively maintain object temporal consistency. To address these challenges, we introduce Ctrl-Adapter, an efficient and versatile framework that adds diverse controls to any image/video diffusion model through the adaptation of pretrained ControlNets. Ctrl-Adapter offers strong and diverse capabilities, including image and video control, sparse-frame video control, fine-grained patch-level multi-condition control (via an MoE router), zero-shot adaptation to unseen conditions, and supports a variety of downstream tasks beyond spatial control, including video editing, video style transfer, and text-guided motion control. With six diverse U-Net/DiT-based image/video diffusion models (SDXL, PixArt-$\alpha$, I2VGen-XL, SVD, Latte, Hotshot-XL), Ctrl-Adapter matches the performance of pretrained ControlNets on COCO and achieves the state-of-the-art on DAVIS 2017 with significantly lower computation (< 10 GPU hours).

翻译：ControlNet 被广泛用于为文本到图像扩散模型添加不同条件（如深度图、涂鸦/草图和人姿态）的空间控制。然而，在可控视频生成领域，由于特征空间不匹配，ControlNet 无法直接集成到新的骨干网络中，而为新骨干网络训练 ControlNet 对许多用户而言可能是一项重大负担。此外，将 ControlNet 独立应用于不同帧无法有效保持对象的时间一致性。为应对这些挑战，我们引入了 Ctrl-Adapter，这是一个高效且通用的框架，它通过适配预训练的 ControlNet，为任意图像/视频扩散模型添加多样化控制。Ctrl-Adapter 提供强大且多样化的能力，包括图像与视频控制、稀疏帧视频控制、细粒度块级多条件控制（通过 MoE 路由器）、对未见条件的零样本适配，并支持超越空间控制的各种下游任务，包括视频编辑、视频风格迁移和文本引导的运动控制。在六种不同的基于 U-Net/DiT 的图像/视频扩散模型（SDXL, PixArt-$\alpha$, I2VGen-XL, SVD, Latte, Hotshot-XL）上，Ctrl-Adapter 在 COCO 数据集上匹配了预训练 ControlNet 的性能，并在 DAVIS 2017 数据集上以显著更低的计算量（< 10 GPU 小时）达到了最先进的水平。