We present a novel inverse rendering-based framework to estimate the 3D shape (per-pixel surface normals and depth) of objects and scenes from single-view polarization images, the problem popularly known as Shape from Polarization (SfP). The existing physics-based and learning-based methods for SfP perform under certain restrictions, i.e., (a) purely diffuse or purely specular reflections, which are seldom in the real surfaces, (b) availability of the ground truth surface normals for direct supervision that are hard to acquire and are limited by the scanner's resolution, and (c) known refractive index. To overcome these restrictions, we start by learning to separate the partially-polarized diffuse and specular reflection components, which we call reflectance cues, based on a modified polarization reflection model and then estimate shape under mixed polarization through an inverse-rendering based self-supervised deep learning framework called SS-SfP, guided by the polarization data and estimated reflectance cues. Furthermore, we also obtain the refractive index as a non-linear least squares solution. Through extensive quantitative and qualitative evaluation, we establish the efficacy of the proposed framework over simple single-object scenes from DeepSfP dataset and complex in-the-wild scenes from SPW dataset in an entirely self-supervised setting. To the best of our knowledge, this is the first learning-based approach to address SfP under mixed polarization in a completely self-supervised framework.

翻译：我们提出了一种新颖的基于逆向渲染的框架，用于从单视角偏振图像中估计物体和场景的三维形状（逐像素表面法线和深度），该问题通常被称为偏振三维形状重建（SfP）。现有的基于物理和基于学习的SfP方法在特定限制下运行，即：（a）仅适用于纯漫反射或纯镜面反射，而这在真实表面中很少见；（b）需要难以获取且受扫描仪分辨率限制的真实表面法线数据进行直接监督；（c）已知折射率。为克服这些限制，我们首先基于改进的偏振反射模型学习分离部分偏振的漫反射和镜面反射分量（我们称之为反射线索），随后通过一个名为SS-SfP的基于逆向渲染的自监督深度学习框架，在偏振数据和估计的反射线索引导下，实现混合偏振条件下的形状估计。此外，我们还通过非线性最小二乘解获得了折射率。通过广泛的定量和定性评估，我们在完全自监督的设置下，验证了所提框架在DeepSfP数据集的简单单物体场景和SPW数据集的复杂真实场景中的有效性。据我们所知，这是首个在完全自监督框架下解决混合偏振条件下SfP问题的基于学习方法。