Novel view synthesis is an essential functionality for enabling immersive experiences in various Augmented- and Virtual-Reality (AR/VR) applications, for which generalizable Neural Radiance Fields (NeRFs) have gained increasing popularity thanks to their cross-scene generalization capability. Despite their promise, the real-device deployment of generalizable NeRFs is bottlenecked by their prohibitive complexity due to the required massive memory accesses to acquire scene features, causing their ray marching process to be memory-bounded. To this end, we propose Gen-NeRF, an algorithm-hardware co-design framework dedicated to generalizable NeRF acceleration, which for the first time enables real-time generalizable NeRFs. On the algorithm side, Gen-NeRF integrates a coarse-then-focus sampling strategy, leveraging the fact that different regions of a 3D scene contribute differently to the rendered pixel, to enable sparse yet effective sampling. On the hardware side, Gen-NeRF highlights an accelerator micro-architecture to maximize the data reuse opportunities among different rays by making use of their epipolar geometric relationship. Furthermore, our Gen-NeRF accelerator features a customized dataflow to enhance data locality during point-to-hardware mapping and an optimized scene feature storage strategy to minimize memory bank conflicts. Extensive experiments validate the effectiveness of our proposed Gen-NeRF framework in enabling real-time and generalizable novel view synthesis.

翻译：新视角合成是实现增强现实/虚拟现实（AR/VR）应用中沉浸式体验的核心功能，其中可泛化神经辐射场（NeRF）因具备跨场景泛化能力而日益受到关注。尽管前景广阔，但可泛化NeRF在真实设备部署中面临严峻挑战：获取场景特征需大量内存访问导致的巨量计算开销，使得其光线行进过程受限于内存带宽。为此，我们提出Gen-NeRF——首个针对可泛化NeRF加速的算法-硬件协同设计框架，实现了实时可泛化神经辐射场。在算法层面，Gen-NeRF引入粗到精聚焦采样策略，利用三维场景不同区域对渲染像素贡献度差异，实现稀疏而高效的采样。在硬件层面，Gen-NeRF设计了一种加速器微架构，通过利用光线间的极线几何关系最大化数据重用机会。此外，该加速器采用定制化数据流以增强点对硬件映射过程中的数据局部性，并优化场景特征存储策略以减少存储体冲突。大量实验验证了Gen-NeRF框架在实现实时可泛化新视角合成方面的有效性。