Quad Bayer demosaicing is the central challenge for enabling the widespread application of Hybrid Event-based Vision Sensors (HybridEVS). Although existing learning-based methods that leverage long-range dependency modeling have achieved promising results, their complexity severely limits deployment on mobile devices for real-world applications. To address these limitations, we propose a lightweight Mamba-based binary neural network designed for efficient and high-performing demosaicing of HybridEVS RAW images. First, to effectively capture both global and local dependencies, we introduce a hybrid Binarized Mamba-Transformer architecture that combines the strengths of the Mamba and Swin Transformer architectures. Next, to significantly reduce computational complexity, we propose a binarized Mamba (Bi-Mamba), which binarizes all projections while retaining the core Selective Scan in full precision. Bi-Mamba also incorporates additional global visual information to enhance global context and mitigate precision loss. We conduct quantitative and qualitative experiments to demonstrate the effectiveness of BMTNet in both performance and computational efficiency, providing a lightweight demosaicing solution suited for real-world edge devices. Our codes and models are available at https://github.com/Clausy9/BMTNet.
翻译:四拜耳去马赛克是实现混合事件视觉传感器广泛应用的核心挑战。尽管现有基于学习的方法通过利用长程依赖建模已取得良好效果,但其复杂性严重限制了在实际移动设备上的部署应用。为应对这些限制,我们提出了一种基于Mamba的轻量化二值神经网络,旨在高效且高性能地完成混合事件视觉传感器RAW图像的去马赛克任务。首先,为有效捕捉全局与局部依赖关系,我们引入了一种混合的二值化Mamba-Transformer架构,该架构融合了Mamba与Swin Transformer的优势。其次,为显著降低计算复杂度,我们提出了二值化Mamba模块,该模块将所有投影操作二值化,同时保留核心的选择性扫描机制为全精度计算。Bi-Mamba还整合了额外的全局视觉信息,以增强全局上下文感知并缓解精度损失。我们通过定量与定性实验验证了BMTNet在性能与计算效率方面的有效性,为实际边缘设备提供了一种轻量化的去马赛克解决方案。我们的代码与模型已开源:https://github.com/Clausy9/BMTNet。