Embedding polygonal mesh assets within photorealistic Neural Radience Fields (NeRF) volumes, such that they can be rendered and their dynamics simulated in a physically consistent manner with the NeRF, is under-explored from the system perspective of integrating NeRF into the traditional graphics pipeline. This paper designs a two-way coupling between mesh and NeRF during rendering and simulation. We first review the light transport equations for both mesh and NeRF, then distill them into an efficient algorithm for updating radiance and throughput along a cast ray with an arbitrary number of bounces. To resolve the discrepancy between the linear color space that the path tracer assumes and the sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range (HDR) images. We also present a strategy to estimate light sources and cast shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric formulation can be efficiently integrated with a high-performance physics simulator that supports cloth, rigid and soft bodies. The full rendering and simulation system can be run on a GPU at interactive rates. We show that a hybrid system approach outperforms alternatives in visual realism for mesh insertion, because it allows realistic light transport from volumetric NeRF media onto surfaces, which affects the appearance of reflective/refractive surfaces and illumination of diffuse surfaces informed by the dynamic scene.

翻译：将多边形网格资产嵌入到逼真的神经辐射场（NeRF）体素中，使其能够以与NeRF物理一致的方式进行渲染和动力学模拟，从将NeRF集成到传统图形管线的系统角度来看，这一问题尚未得到充分探索。本文设计了渲染和模拟过程中网格与NeRF之间的双向耦合。我们首先回顾了网格和NeRF的光传输方程，然后将其提炼为一种高效算法，该算法可沿一条具有任意弹跳次数的投射光线更新辐射度和通量。为了解决路径追踪器假设的线性色彩空间与标准NeRF使用的sRGB色彩空间之间的差异，我们使用高动态范围（HDR）图像训练NeRF。我们还提出了一种估计光源并在NeRF上投射阴影的策略。最后，我们考虑了如何将混合表面-体积公式高效集成到支持布料、刚体和软体的高性能物理模拟器中。完整的渲染和模拟系统可在GPU上以交互速率运行。我们表明，混合系统方法在网格插入的视觉真实感方面优于替代方案，因为它允许来自体积NeRF介质的逼真光传输到表面上，这影响了反射/折射表面的外观以及动态场景所决定的漫射表面照明。