Digital cameras consume ~0.1 microjoule per pixel to capture and encode video, resulting in a power usage of ~20W for a 4K sensor operating at 30 fps. Imagining gigapixel cameras operating at 100-1000 fps, the current processing model is unsustainable. To address this, physical layer compressive measurement has been proposed to reduce power consumption per pixel by 10-100X. Video Snapshot Compressive Imaging (SCI) introduces high frequency modulation in the optical sensor layer to increase effective frame rate. A commonly used sampling strategy of video SCI is Random Sampling (RS) where each mask element value is randomly set to be 0 or 1. Similarly, image inpainting (I2P) has demonstrated that images can be recovered from a fraction of the image pixels. Inspired by I2P, we propose Ultra-Sparse Sampling (USS) regime, where at each spatial location, only one sub-frame is set to 1 and all others are set to 0. We then build a Digital Micro-mirror Device (DMD) encoding system to verify the effectiveness of our USS strategy. Ideally, we can decompose the USS measurement into sub-measurements for which we can utilize I2P algorithms to recover high-speed frames. However, due to the mismatch between the DMD and CCD, the USS measurement cannot be perfectly decomposed. To this end, we propose BSTFormer, a sparse TransFormer that utilizes local Block attention, global Sparse attention, and global Temporal attention to exploit the sparsity of the USS measurement. Extensive results on both simulated and real-world data show that our method significantly outperforms all previous state-of-the-art algorithms. Additionally, an essential advantage of the USS strategy is its higher dynamic range than that of the RS strategy. Finally, from the application perspective, the USS strategy is a good choice to implement a complete video SCI system on chip due to its fixed exposure time.

翻译：数码相机每像素约消耗0.1微焦耳的能量来捕获和编码视频，导致一个以30帧/秒运行的4K传感器功耗约为20W。设想以100-1000帧/秒运行的千兆像素相机，当前的处理模式是不可持续的。为解决此问题，已提出物理层压缩测量技术，可将每像素功耗降低10-100倍。视频快照压缩成像（SCI）在光学传感器层引入高频调制以提高有效帧率。视频SCI中一种常用的采样策略是随机采样（RS），其中每个掩模元素值被随机设置为0或1。类似地，图像修复（I2P）已证明可以从部分图像像素中恢复图像。受I2P启发，我们提出超稀疏采样（USS）机制，其中在每个空间位置上，仅有一个子帧被设置为1，其余所有子帧均设置为0。随后，我们构建了一个数字微镜器件（DMD）编码系统来验证USS策略的有效性。理想情况下，我们可以将USS测量分解为子测量，并利用I2P算法来恢复高速帧。然而，由于DMD与CCD之间的不匹配，USS测量无法被完美分解。为此，我们提出BSTFormer，一种稀疏Transformer，它利用局部块注意力、全局稀疏注意力和全局时序注意力来挖掘USS测量的稀疏性。在仿真和真实数据上的大量结果表明，我们的方法显著优于所有先前的最先进算法。此外，USS策略的一个关键优势是其动态范围高于RS策略。最后，从应用角度看，由于其固定的曝光时间，USS策略是实现片上完整视频SCI系统的良好选择。