Visual error metrics play a fundamental role in the quantification of perceived image similarity. Most recently, use cases for them in real-time applications have emerged, such as content-adaptive shading and shading reuse to increase performance and improve efficiency. A wide range of different metrics has been established, with the most sophisticated being capable of capturing the perceptual characteristics of the human visual system. However, their complexity, computational expense, and reliance on reference images to compare against prevent their generalized use in real-time, restricting such applications to using only the simplest available metrics. In this work, we explore the abilities of convolutional neural networks to predict a variety of visual metrics without requiring either reference or rendered images. Specifically, we train and deploy a neural network to estimate the visual error resulting from reusing shading or using reduced shading rates. The resulting models account for 70%-90% of the variance while achieving up to an order of magnitude faster computation times. Our solution combines image-space information that is readily available in most state-of-the-art deferred shading pipelines with reprojection from previous frames to enable an adequate estimate of visual errors, even in previously unseen regions. We describe a suitable convolutional network architecture and considerations for data preparation for training. We demonstrate the capability of our network to predict complex error metrics at interactive rates in a real-time application that implements content-adaptive shading in a deferred pipeline. Depending on the portion of unseen image regions, our approach can achieve up to $2\times$ performance compared to state-of-the-art methods.

翻译：视觉误差度量在量化感知图像相似度方面起着基础性作用。近年来，实时应用中的视觉误差度量使用场景不断涌现，例如内容自适应着色和着色重用，以提升性能与效率。目前已有多种不同的度量标准被提出，其中最为复杂的指标能够捕捉人类视觉系统的感知特性。然而，这些度量指标的复杂性、高计算开销以及对参考图像进行比对的需求，阻碍了其在实时场景中的普遍应用，导致此类应用仅能使用最简单的可用度量指标。本研究探索了卷积神经网络在不依赖参考图像或渲染图像的情况下预测多种视觉度量指标的能力。具体而言，我们训练并部署了一个神经网络，用于估算因着色重用或降低着色频率而产生的视觉误差。由此生成的模型能够解释70%至90%的方差，同时计算速度提升最高可达一个数量级。我们的解决方案结合了大多数先进延迟着色管线中易于获取的图像空间信息，以及来自先前帧的重投影，从而能够对视觉误差进行充分估计，即使是针对此前未见过的区域。我们描述了适用的卷积网络架构及其数据准备注意事项。通过一个在延迟管线中实现内容自适应着色的实时应用，我们展示了网络以交互速率预测复杂误差度量的能力。根据未见图像区域的比例，我们的方法相比现有先进技术可实现高达两倍的性能提升。