Event cameras with high dynamic range ensure scene capture even in low-light conditions. However, night events exhibit patterns different from those captured during the day. This difference causes performance degradation when applying night events to a model trained solely on day events. This limitation persists due to a lack of annotated night events. To overcome the limitation, we aim to alleviate data imbalance by translating annotated day data into night events. However, generating events from different modalities challenges reproducing their unique properties. Accordingly, we propose an unpaired event-to-event day-to-night translation model that effectively learns to map from one domain to another using Diffusion GAN. The proposed translation model analyzes events in spatio-temporal dimension with wavelet decomposition and disentangled convolution layers. We also propose a new temporal contrastive learning with a novel shuffling and sampling strategy to regularize temporal continuity. To validate the efficacy of the proposed methodology, we redesign metrics for evaluating events translated in an unpaired setting, aligning them with the event modality for the first time. Our framework shows the successful day-to-night event translation while preserving the characteristics of events. In addition, through our translation method, we facilitate event-based modes to learn about night events by translating annotated day events into night events. Our approach effectively mitigates the performance degradation of applying real night events to downstream tasks. The code is available at https://github.com/jeongyh98/UDNET.
翻译:具备高动态范围的事件相机即使在低光照条件下也能确保场景捕捉。然而,夜间事件呈现出与日间捕获事件不同的模式。当将夜间事件应用于仅使用日间事件训练的模型时,这种差异会导致性能下降。由于缺乏标注的夜间事件数据,这一局限性持续存在。为克服此限制,我们旨在通过将标注的日间数据转换为夜间事件来缓解数据不平衡问题。然而,从不同模态生成事件面临着再现其独特属性的挑战。为此,我们提出了一种非配对事件到事件的日间至夜间转换模型,该模型利用扩散生成对抗网络(Diffusion GAN)有效学习从一个域到另一个域的映射。所提出的转换模型通过小波分解与解耦卷积层在时空维度上分析事件。我们还提出了一种新的时序对比学习方法,采用创新的重排与采样策略以规范时序连续性。为验证所提方法的有效性,我们重新设计了用于评估非配对设置下转换事件的指标,首次使其与事件模态特性相匹配。我们的框架在保持事件特征的同时,成功实现了日间到夜间的事件转换。此外,通过我们的转换方法,我们通过将标注的日间事件转换为夜间事件,促进了基于事件的模型学习夜间事件。我们的方法有效缓解了将真实夜间事件应用于下游任务时的性能下降问题。代码发布于 https://github.com/jeongyh98/UDNET。