While recent works on blind face image restoration have successfully produced impressive high-quality (HQ) images with abundant details from low-quality (LQ) input images, the generated content may not accurately reflect the real appearance of a person. To address this problem, incorporating well-shot personal images as additional reference inputs could be a promising strategy. Inspired by the recent success of the Latent Diffusion Model (LDM), we propose ReF-LDM, an adaptation of LDM designed to generate HQ face images conditioned on one LQ image and multiple HQ reference images. Our model integrates an effective and efficient mechanism, CacheKV, to leverage the reference images during the generation process. Additionally, we design a timestep-scaled identity loss, enabling our LDM-based model to focus on learning the discriminating features of human faces. Lastly, we construct FFHQ-Ref, a dataset consisting of 20,405 high-quality (HQ) face images with corresponding reference images, which can serve as both training and evaluation data for reference-based face restoration models.

翻译：尽管近期关于盲人脸图像复原的研究已成功地从低质量输入图像中生成出细节丰富、令人印象深刻的高质量图像，但生成的内容可能无法准确反映人物的真实外貌。为解决此问题，引入精心拍摄的个人图像作为额外的参考输入可能是一种有前景的策略。受近期潜在扩散模型成功的启发，我们提出了ReF-LDM，这是一种经过改进的LDM，旨在基于一张低质量图像和多张高质量参考图像生成高质量人脸图像。我们的模型集成了一个高效且有效的机制——CacheKV，以在生成过程中充分利用参考图像。此外，我们设计了一种时间步缩放的身份损失函数，使我们基于LDM的模型能够专注于学习人脸的可区分特征。最后，我们构建了FFHQ-Ref数据集，该数据集包含20,405张高质量人脸图像及其对应的参考图像，可作为基于参考的人脸复原模型的训练和评估数据。