Feature embeddings acquired from pretrained models are widely used in medical applications of deep learning to assess the characteristics of datasets; e.g. to determine the quality of synthetic, generated medical images. The Fréchet Inception Distance (FID) is one popular synthetic image quality metric that relies on the assumption that the characteristic features of the data can be detected and encoded by an InceptionV3 model pretrained on ImageNet1K (natural images). While it is widely known that this makes it less effective for applications involving medical images, the extent to which the metric fails to capture meaningful differences in image characteristics is not obviously known. Here, we use Monte Carlo dropout to compute the predictive variance in the FID as well as a supplemental estimate of the predictive variance in the feature embedding model's latent representations. We show that the magnitudes of the predictive variances considered exhibit varying degrees of correlation with the extent to which test inputs (ImageNet1K validation set augmented at various strengths, and other external datasets) are out-of-distribution relative to its training data, providing some insight into the effectiveness of their use as indicators of the trustworthiness of the FID.

翻译：从预训练模型获取的特征嵌入在深度学习的医学应用中广泛用于评估数据集特性，例如判断合成生成的医学图像质量。Fréchet Inception Distance（FID）是一种常用的合成图像质量评估指标，其前提假设是数据的特征能够被在ImageNet1K（自然图像）上预训练的InceptionV3模型检测并编码。尽管众所周知这使得该指标在处理医学图像时效果受限，但该度量标准在多大程度上未能捕捉图像特征的有意义差异尚不明确。本研究采用蒙特卡洛dropout方法计算FID的预测方差，并补充估计特征嵌入模型潜在表示的预测方差。我们发现，所考察的预测方差幅度与测试输入（经不同强度增强的ImageNet1K验证集及其他外部数据集）相对于训练数据的分布外偏移程度呈现不同程度的相关性，这为将其作为FID可信度指示器的有效性提供了新的见解。