Video frame interpolation (VFI) enables many important applications that might involve the temporal domain, such as slow motion playback, or the spatial domain, such as stop motion sequences. We are focusing on the former task, where one of the key challenges is handling high dynamic range (HDR) scenes in the presence of complex motion. To this end, we explore possible advantages of dual-exposure sensors that readily provide sharp short and blurry long exposures that are spatially registered and whose ends are temporally aligned. This way, motion blur registers temporally continuous information on the scene motion that, combined with the sharp reference, enables more precise motion sampling within a single camera shot. We demonstrate that this facilitates a more complex motion reconstruction in the VFI task, as well as HDR frame reconstruction that so far has been considered only for the originally captured frames, not in-between interpolated frames. We design a neural network trained in these tasks that clearly outperforms existing solutions. We also propose a metric for scene motion complexity that provides important insights into the performance of VFI methods at the test time.

翻译：视频帧插值（VFI）可实现涉及时间域（如慢动作播放）或空间域（如停格动画）的诸多重要应用。本文聚焦于前一类任务，其中关键挑战之一是在存在复杂运动的情况下处理高动态范围（HDR）场景。为此，我们探索了双曝光传感器的潜在优势，该传感器可提供空间对齐且时间端点对齐的清晰短曝光图像与模糊长曝光图像。通过这种方式，运动模糊记录了场景运动的连续时间信息，结合清晰的参考帧，可在单次相机拍摄中实现更精确的运动采样。我们证明，这种方法有助于在VFI任务中实现更复杂的运动重建，以及HDR帧重建——此前HDR重建仅针对原始捕获帧，而非插值中间帧。我们设计了针对这些任务训练的神经网络，其性能明显优于现有方案。我们还提出了一种场景运动复杂度度量，为测试时VFI方法的性能评估提供了重要见解。