Novel view synthesis is an essential functionality for enabling immersive experiences in various Augmented- and Virtual-Reality (AR/VR) applications, for which generalizable Neural Radiance Fields (NeRFs) have gained increasing popularity thanks to their cross-scene generalization capability. Despite their promise, the real-device deployment of generalizable NeRFs is bottlenecked by their prohibitive complexity due to the required massive memory accesses to acquire scene features, causing their ray marching process to be memory-bounded. To this end, we propose Gen-NeRF, an algorithm-hardware co-design framework dedicated to generalizable NeRF acceleration, which for the first time enables real-time generalizable NeRFs. On the algorithm side, Gen-NeRF integrates a coarse-then-focus sampling strategy, leveraging the fact that different regions of a 3D scene contribute differently to the rendered pixel, to enable sparse yet effective sampling. On the hardware side, Gen-NeRF highlights an accelerator micro-architecture to maximize the data reuse opportunities among different rays by making use of their epipolar geometric relationship. Furthermore, our Gen-NeRF accelerator features a customized dataflow to enhance data locality during point-to-hardware mapping and an optimized scene feature storage strategy to minimize memory bank conflicts. Extensive experiments validate the effectiveness of our proposed Gen-NeRF framework in enabling real-time and generalizable novel view synthesis.

翻译：新视角合成是增强现实和虚拟现实（AR/VR）应用中实现沉浸式体验的关键功能，其中可泛化神经辐射场（NeRF）因其跨场景泛化能力而日益受到关注。尽管前景广阔，但可泛化NeRF在实际设备中的部署受限于其高昂的计算复杂度——获取场景特征所需的大量内存访问导致光线行进过程受限于内存带宽。为解决此问题，我们提出Gen-NeRF，这是首个专用于可泛化NeRF加速的算法-硬件协同设计框架，首次实现了实时可泛化NeRF。在算法层面，Gen-NeRF整合了一种粗到精的采样策略，利用三维场景不同区域对渲染像素贡献度不同的特性，实现稀疏但高效的采样。在硬件层面，Gen-NeRF提出了一种加速器微架构，通过利用光线间的极线几何关系最大化数据重用机会。此外，我们的Gen-NeRF加速器包含定制化的数据流以增强点-硬件映射过程中的数据局部性，以及优化场景特征存储策略以最小化存储体冲突。大量实验验证了所提出的Gen-NeRF框架在实现实时且可泛化的新视角合成方面的有效性。