We propose a hybrid recurrent Video Colorization with Hybrid Generative Adversarial Network (VCGAN), an improved approach to video colorization using end-to-end learning. The VCGAN addresses two prevalent issues in the video colorization domain: Temporal consistency and unification of colorization network and refinement network into a single architecture. To enhance colorization quality and spatiotemporal consistency, the mainstream of generator in VCGAN is assisted by two additional networks, i.e., global feature extractor and placeholder feature extractor, respectively. The global feature extractor encodes the global semantics of grayscale input to enhance colorization quality, whereas the placeholder feature extractor acts as a feedback connection to encode the semantics of the previous colorized frame in order to maintain spatiotemporal consistency. If changing the input for placeholder feature extractor as grayscale input, the hybrid VCGAN also has the potential to perform image colorization. To improve the consistency of far frames, we propose a dense long-term loss that smooths the temporal disparity of every two remote frames. Trained with colorization and temporal losses jointly, VCGAN strikes a good balance between color vividness and video continuity. Experimental results demonstrate that VCGAN produces higher-quality and temporally more consistent colorful videos than existing approaches.

翻译：我们提出了一种混合递归视频彩色化方法——VCGAN（Video Colorization with Hybrid Generative Adversarial Network），这是一种通过端到端学习改进视频彩色化的方法。VCGAN解决了视频彩色化领域的两个普遍问题：时间一致性问题，以及将彩色化网络与细化网络统一为单一架构的问题。为了提升彩色化质量与时空一致性，VCGAN生成器的主干网络分别由两个辅助网络（即全局特征提取器和占位符特征提取器）提供支持。全局特征提取器对灰度输入的全局语义进行编码以增强彩色化质量，而占位符特征提取器则作为反馈连接，对前一帧彩色化结果的语义进行编码，以维持时空一致性。若将占位符特征提取器的输入替换为灰度输入，混合结构VCGAN还可用于图像彩色化。为改善远距离帧的一致性，我们提出了一种密集长期损失函数，用于平滑任意两帧间的时间差异。通过联合训练彩色化损失与时间损失，VCGAN在色彩鲜明度与视频连续性之间取得了良好平衡。实验结果表明，VCGAN生成的彩色视频在质量和时间一致性上均优于现有方法。