We present NeuriCam, a novel deep learning-based system to achieve video capture from low-power dual-mode IoT camera systems. Our idea is to design a dual-mode camera system where the first mode is low-power (1.1 mW) but only outputs grey-scale, low resolution, and noisy video and the second mode consumes much higher power (100 mW) but outputs color and higher resolution images. To reduce total energy consumption, we heavily duty cycle the high power mode to output an image only once every second. The data for this camera system is then wirelessly sent to a nearby plugged-in gateway, where we run our real-time neural network decoder to reconstruct a higher-resolution color video. To achieve this, we introduce an attention feature filter mechanism that assigns different weights to different features, based on the correlation between the feature map and the contents of the input frame at each spatial location. We design a wireless hardware prototype using off-the-shelf cameras and address practical issues including packet loss and perspective mismatch. Our evaluations show that our dual-camera approach reduces energy consumption by 7x compared to existing systems. Further, our model achieves an average greyscale PSNR gain of 3.7 dB over prior single and dual-camera video super-resolution methods and 5.6 dB RGB gain over prior color propagation methods. Open-source code: https://github.com/vb000/NeuriCam.

翻译：我们提出NeuriCam，一种基于深度学习的新型系统，用于从低功耗双模物联网相机系统捕获视频。其核心思想是设计一个双模相机系统：第一模式功耗低（1.1 mW），但仅输出灰度、低分辨率且含噪视频；第二模式功耗较高（100 mW），但输出彩色且更高分辨率图像。为降低总能耗，我们采用高占空比控制高功耗模式，使其每秒仅输出一帧图像。相机系统采集的数据通过无线方式传输至附近接入式网关，并在其上运行实时神经网络解码器，以重建高分辨率彩色视频。为此，我们引入一种注意力特征过滤机制，根据特征图与输入帧内容在每个空间位置的相关性，为不同特征赋予不同权重。我们利用现成摄像头设计了无线硬件原型，并解决了数据包丢失和视角不匹配等实际问题。评估表明，与现有系统相比，我们的双相机方法将能耗降低7倍。此外，与先前的单相机和双相机视频超分辨率方法相比，模型在灰度PSNR上平均提升3.7 dB；与先前的颜色传播方法相比，在RGB增益上提升5.6 dB。开源代码：https://github.com/vb000/NeuriCam。