Forensic sketch-to-mugshot matching is a challenging task in face recognition, primarily hindered by the scarcity of annotated forensic sketches and the modality gap between sketches and photographs. To address this, we propose CLIP4Sketch, a novel approach that leverages diffusion models to generate a large and diverse set of sketch images, which helps in enhancing the performance of face recognition systems in sketch-to-mugshot matching. Our method utilizes Denoising Diffusion Probabilistic Models (DDPMs) to generate sketches with explicit control over identity and style. We combine CLIP and Adaface embeddings of a reference mugshot, along with textual descriptions of style, as the conditions to the diffusion model. We demonstrate the efficacy of our approach by generating a comprehensive dataset of sketches corresponding to mugshots and training a face recognition model on our synthetic data. Our results show significant improvements in sketch-to-mugshot matching accuracy over training on an existing, limited amount of real face sketch data, validating the potential of diffusion models in enhancing the performance of face recognition systems across modalities. We also compare our dataset with datasets generated using GAN-based methods to show its superiority.

翻译：法医素描与面部照片匹配是人脸识别领域的一项挑战性任务，其主要障碍在于标注的法医素描数据稀缺以及素描与照片之间的模态差异。为解决此问题，我们提出CLIP4Sketch，一种利用扩散模型生成大规模多样化素描图像集的新方法，以提升人脸识别系统在素描与面部照片匹配任务中的性能。我们的方法采用去噪扩散概率模型（DDPMs）生成素描，并能对身份与风格进行显式控制。我们将参考面部照片的CLIP嵌入与Adaface嵌入，连同风格文本描述，共同作为扩散模型的条件输入。我们通过生成与面部照片对应的大规模素描数据集，并基于合成数据训练人脸识别模型，验证了该方法的有效性。实验结果表明，相较于在现有有限真实人脸素描数据上训练，我们的方法在素描与面部照片匹配准确率上取得了显著提升，证实了扩散模型在增强跨模态人脸识别系统性能方面的潜力。我们还通过将所生成数据集与基于GAN方法生成的数据集进行比较，展示了其优越性。