We propose LookinGood^{\pi}, a novel neural re-rendering approach that is aimed to (1) improve the rendering quality of the low-quality reconstructed results from human performance capture system in real-time; (2) improve the generalization ability of the neural rendering network on unseen people. Our key idea is to utilize the rendered image of reconstructed geometry as the guidance to assist the prediction of person-specific details from few reference images, thus enhancing the re-rendered result. In light of this, we design a two-branch network. A coarse branch is designed to fix some artifacts (i.e. holes, noise) and obtain a coarse version of the rendered input, while a detail branch is designed to predict "correct" details from the warped references. The guidance of the rendered image is realized by blending features from two branches effectively in the training of the detail branch, which improves both the warping accuracy and the details' fidelity. We demonstrate that our method outperforms state-of-the-art methods at producing high-fidelity images on unseen people.

翻译：我们提出LookinGood^{\pi}，一种新颖的神经重渲染方法，旨在：(1) 实时提升人体性能捕捉系统中低质量重建结果的渲染质量；(2) 提高神经渲染网络对未见人物的泛化能力。核心思想是利用重建几何体的渲染图像作为引导，从少量参考图像中辅助预测人物特定细节，从而增强重渲染结果。据此，我们设计了一个双分支网络：粗糙分支用于修复伪影（如空洞、噪声）并获取渲染输入的粗略版本，而细节分支则用于从变形参考图像中预测"正确"细节。渲染图像的引导通过细节分支训练中有效融合两个分支的特征来实现，这同时提升了变形精度与细节保真度。实验表明，我们的方法在生成未见人物的高保真图像方面优于当前最先进方法。