Rendering high-resolution images in real-time applications (e.g., video games, virtual reality) is time-consuming, thus super-resolution technology becomes more and more crucial in real-time rendering. However, it is still challenging to preserve sharp texture details, keep the temporal stability and avoid the ghosting artifacts in the real-time rendering super-resolution. To this end, we introduce radiance demodulation into real-time rendering super-resolution, separating the rendered image or radiance into a lighting component and a material component, due to the fact that the light component tends to be smoother than the rendered image and the high-resolution material component with detailed textures can be easily obtained. Therefore, we perform the super-resolution only on the lighting component and re-modulate with the high-resolution material component to obtain the final super-resolution image. In this way, the texture details can be preserved much better. Then, we propose a reliable warping module by explicitly pointing out the unreliable occluded regions with a motion mask to remove the ghosting artifacts. We further enhance the temporal stability by designing a frame-recurrent neural network to aggregate the previous and current frames, which better captures the spatial-temporal correlation between reconstructed frames. As a result, our method is able to produce temporally stable results in real-time rendering with high-quality details, even in the highly challenging 4 $\times$ 4 super-resolution scenarios.

翻译：摘要：在实时应用（如电子游戏、虚拟现实）中渲染高分辨率图像耗时巨大，因此超分辨率技术在实时渲染领域变得愈发重要。然而，在实时渲染超分辨率过程中，保持锐利纹理细节、维持时间稳定性并避免鬼影伪影仍是重大挑战。为此，本文提出将辐射解调引入实时渲染超分辨率，根据光照分量通常比渲染图像更平滑、而高分辨率材质分量容易获取纹理细节的特性，将渲染图像或辐射分解为光照分量与材质分量。我们仅对光照分量执行超分辨率重建，再与高分辨率材质分量重新调制，最终获得超分辨率图像。该方法能显著提升纹理细节的保持效果。同时，我们通过运动掩膜显式标记不可靠遮挡区域，提出可靠的光流扭曲模块以消除鬼影伪影。进一步设计帧循环神经网络聚合前后帧信息，增强时间稳定性，更好地捕捉重建帧之间的时空关联。实验表明，我们的方法能在实时渲染中生成具有高质量细节的时间稳定结果，即使在4×4倍超分辨率的高挑战性场景下依然表现优异。