We systematically study a wide variety of image-based generative models spanning semantically-diverse datasets to understand and improve the feature extractors and metrics used to evaluate them. Using best practices in psychophysics, we measure human perception of image realism for generated samples by conducting the largest experiment evaluating generative models to date, and find that no existing metric strongly correlates with human evaluations. Comparing to 16 modern metrics for evaluating the overall performance, fidelity, diversity, and memorization of generative models, we find that the state-of-the-art perceptual realism of diffusion models as judged by humans is not reflected in commonly reported metrics such as FID. This discrepancy is not explained by diversity in generated samples, though one cause is over-reliance on Inception-V3. We address these flaws through a study of alternative self-supervised feature extractors, find that the semantic information encoded by individual networks strongly depends on their training procedure, and show that DINOv2-ViT-L/14 allows for much richer evaluation of generative models. Next, we investigate data memorization, and find that generative models do memorize training examples on simple, smaller datasets like CIFAR10, but not necessarily on more complex datasets like ImageNet. However, our experiments show that current metrics do not properly detect memorization; none in the literature is able to separate memorization from other phenomena such as underfitting or mode shrinkage. To facilitate further development of generative models and their evaluation we release all generated image datasets, human evaluation data, and a modular library to compute 16 common metrics for 8 different encoders at https://github.com/layer6ai-labs/dgm-eval.

翻译：我们系统性地研究了涵盖语义多样化数据集的各类基于图像的生成模型，以理解并改进用于评估它们的特征提取器和指标。通过应用心理物理学中的最佳实践，我们开展了迄今最大规模的生成模型评估实验，测量了人类对生成样本图像真实性的感知，并发现现有指标与人类评估结果均无强相关性。在对比用于评估生成模型整体性能、保真度、多样性及记忆化程度的16种现代指标后，我们发现人类判定的扩散模型最先进感知真实性在FID等常用指标中并未得到体现。这种差异无法通过生成样本的多样性来解释，但部分原因在于对Inception-V3的过度依赖。我们通过研究替代性自监督特征提取器来解决这些缺陷，发现单个网络编码的语义信息强烈取决于其训练过程，并证明DINOv2-ViT-L/14能够对生成模型进行更丰富的评估。接着，我们研究了数据记忆化现象，发现生成模型确实会在CIFAR10等简单小型数据集上记忆训练样本，但在ImageNet等更复杂数据集上则不一定。然而，我们的实验表明，当前指标无法正确检测记忆化；现有文献中没有任何指标能够将记忆化与欠拟合或模式坍缩等其他现象区分开来。为促进生成模型及其评估的进一步发展，我们在https://github.com/layer6ai-labs/dgm-eval 上发布了所有生成的图像数据集、人类评估数据以及一个用于计算8种不同编码器的16种常用指标的模块化库。