Event cameras harness advantages such as low latency, high temporal resolution, and high dynamic range (HDR), compared to standard cameras. Due to the distinct imaging paradigm shift, a dominant line of research focuses on event-to-video (E2V) reconstruction to bridge event-based and standard computer vision. However, this task remains challenging due to its inherently ill-posed nature: event cameras only detect the edge and motion information locally. Consequently, the reconstructed videos are often plagued by artifacts and regional blur, primarily caused by the ambiguous semantics of event data. In this paper, we find language naturally conveys abundant semantic information, rendering it stunningly superior in ensuring semantic consistency for E2V reconstruction. Accordingly, we propose a novel framework, called LaSe-E2V, that can achieve semantic-aware high-quality E2V reconstruction from a language-guided perspective, buttressed by the text-conditional diffusion models. However, due to diffusion models' inherent diversity and randomness, it is hardly possible to directly apply them to achieve spatial and temporal consistency for E2V reconstruction. Thus, we first propose an Event-guided Spatiotemporal Attention (ESA) module to condition the event data to the denoising pipeline effectively. We then introduce an event-aware mask loss to ensure temporal coherence and a noise initialization strategy to enhance spatial consistency. Given the absence of event-text-video paired data, we aggregate existing E2V datasets and generate textual descriptions using the tagging models for training and evaluation. Extensive experiments on three datasets covering diverse challenging scenarios (e.g., fast motion, low light) demonstrate the superiority of our method.

翻译：与标准相机相比，事件相机具备低延迟、高时间分辨率和高动态范围（HDR）等优势。由于成像范式的根本性转变，当前研究的一个主流方向聚焦于事件到视频（E2V）重建，以弥合基于事件的视觉与标准计算机视觉之间的鸿沟。然而，由于该任务本质上具有不适定性——事件相机仅能局部检测边缘和运动信息——重建视频常受伪影和区域模糊的困扰，这主要源于事件数据语义的模糊性。本文发现，语言天然承载丰富的语义信息，在确保E2V重建的语义一致性方面展现出惊人的优越性。据此，我们提出一个名为LaSe-E2V的新框架，该框架能够从语言引导的视角，借助文本条件扩散模型的支持，实现语义感知的高质量E2V重建。然而，由于扩散模型固有的多样性和随机性，直接将其应用于E2V重建以实现时空一致性极为困难。因此，我们首先提出一个事件引导的时空注意力（ESA）模块，以将事件数据有效地注入去噪流程。接着，我们引入一种事件感知掩码损失来确保时间连贯性，以及一种噪声初始化策略来增强空间一致性。鉴于缺乏事件-文本-视频配对数据，我们整合了现有的E2V数据集，并利用标注模型生成文本描述用于训练和评估。在涵盖多种挑战性场景（如快速运动、低光照）的三个数据集上进行的大量实验证明了我们方法的优越性。