Video colorization is a challenging task that involves inferring plausible and temporally consistent colors for grayscale frames. In this paper, we present ColorDiffuser, an adaptation of a pre-trained text-to-image latent diffusion model for video colorization. With the proposed adapter-based approach, we repropose the pre-trained text-to-image model to accept input grayscale video frames, with the optional text description, for video colorization. To enhance the temporal coherence and maintain the vividness of colorization across frames, we propose two novel techniques: the Color Propagation Attention and Alternated Sampling Strategy. Color Propagation Attention enables the model to refine its colorization decision based on a reference latent frame, while Alternated Sampling Strategy captures spatiotemporal dependencies by using the next and previous adjacent latent frames alternatively as reference during the generative diffusion sampling steps. This encourages bidirectional color information propagation between adjacent video frames, leading to improved color consistency across frames. We conduct extensive experiments on benchmark datasets, and the results demonstrate the effectiveness of our proposed framework. The evaluations show that ColorDiffuser achieves state-of-the-art performance in video colorization, surpassing existing methods in terms of color fidelity, temporal consistency, and visual quality.

翻译：视频着色是一项具有挑战性的任务，涉及为灰度帧推断合理且时间一致的色彩。本文提出ColorDiffuser，一种基于预训练文生图潜在扩散模型的视频着色适配方法。通过所提出的适配器方案，我们重新调整预训练文生图模型以接受输入灰度视频帧（可选文本描述）进行视频着色。为增强帧间时间连贯性并保持着色鲜活性，我们提出两种新技术：色彩传播注意力与交替采样策略。色彩传播注意力使模型能基于参考潜在帧优化着色决策，而交替采样策略通过在生成扩散采样步骤中交替使用相邻的前后潜在帧作为参考，捕捉时空依赖性。该方法促进相邻视频帧间的双向色彩信息传播，从而提升帧间色彩一致性。我们在基准数据集上进行广泛实验，结果验证了所提框架的有效性。评估显示，ColorDiffuser在视频着色中达到最优性能，在色彩保真度、时间一致性和视觉质量方面均超越现有方法。