Recent advances in text-to-video generation have demonstrated the utility of powerful diffusion models. Nevertheless, the problem is not trivial when shaping diffusion models to animate static image (i.e., image-to-video generation). The difficulty originates from the aspect that the diffusion process of subsequent animated frames should not only preserve the faithful alignment with the given image but also pursue temporal coherence among adjacent frames. To alleviate this, we present TRIP, a new recipe of image-to-video diffusion paradigm that pivots on image noise prior derived from static image to jointly trigger inter-frame relational reasoning and ease the coherent temporal modeling via temporal residual learning. Technically, the image noise prior is first attained through one-step backward diffusion process based on both static image and noised video latent codes. Next, TRIP executes a residual-like dual-path scheme for noise prediction: 1) a shortcut path that directly takes image noise prior as the reference noise of each frame to amplify the alignment between the first frame and subsequent frames; 2) a residual path that employs 3D-UNet over noised video and static image latent codes to enable inter-frame relational reasoning, thereby easing the learning of the residual noise for each frame. Furthermore, both reference and residual noise of each frame are dynamically merged via attention mechanism for final video generation. Extensive experiments on WebVid-10M, DTDB and MSR-VTT datasets demonstrate the effectiveness of our TRIP for image-to-video generation. Please see our project page at https://trip-i2v.github.io/TRIP/.

翻译：近期文本到视频生成领域的进展展示了强扩散模型的实用性。然而，将扩散模型应用于静态图像动画化（即图像到视频生成）这一问题时并非易事。其难点在于后续动画帧的扩散过程不仅需保持与给定图像的忠实对齐，还需追求相邻帧间的时间一致性。为解决此问题，我们提出TRIP——一种基于图像噪声先验的图像到视频扩散范式新方案。该方法利用从静态图像中提取的图像噪声先验，通过时间残差学习联合触发帧间关系推理，并简化连贯的时间建模。技术上，首先基于静态图像和带噪视频隐编码，通过一步反向扩散过程获取图像噪声先验。接着，TRIP执行类似残差的双路径方案进行噪声预测：1）捷径路径，直接以图像噪声先验作为每帧的参考噪声，增强首帧与后续帧的对齐；2）残差路径，在带噪视频和静态图像隐编码上采用3D-UNet实现帧间关系推理，从而简化每帧残差噪声的学习。此外，每帧的参考噪声和残差噪声通过注意力机制动态融合以生成最终视频。在WebVid-10M、DTDB和MSR-VTT数据集上的大量实验证明了我们的TRIP在图像到视频生成中的有效性。请参阅我们的项目页面：https://trip-i2v.github.io/TRIP/。