Likelihood-based deep generative models such as score-based diffusion models and variational autoencoders are state-of-the-art machine learning models approximating high-dimensional distributions of data such as images, text, or audio. One of many downstream tasks they can be naturally applied to is out-of-distribution (OOD) detection. However, seminal work by Nalisnick et al. which we reproduce showed that deep generative models consistently infer higher log-likelihoods for OOD data than data they were trained on, marking an open problem. In this work, we analyse using the gradient of a data point with respect to the parameters of the deep generative model for OOD detection, based on the simple intuition that OOD data should have larger gradient norms than training data. We formalise measuring the size of the gradient as approximating the Fisher information metric. We show that the Fisher information matrix (FIM) has large absolute diagonal values, motivating the use of chi-square distributed, layer-wise gradient norms as features. We combine these features to make a simple, model-agnostic and hyperparameter-free method for OOD detection which estimates the joint density of the layer-wise gradient norms for a given data point. We find that these layer-wise gradient norms are weakly correlated, rendering their combined usage informative, and prove that the layer-wise gradient norms satisfy the principle of (data representation) invariance. Our empirical results indicate that this method outperforms the Typicality test for most deep generative models and image dataset pairings.

翻译：基于似然的深度生成模型，如基于分数的扩散模型和变分自编码器，是近似图像、文本或音频等高维数据分布的最先进机器学习模型。它们可自然应用于众多下游任务之一便是分布外（OOD）检测。然而，我们复现了Nalisnick等人的开创性工作，表明深度生成模型始终对OOD数据推断出比其训练数据更高的对数似然，这标志着一个悬而未决的问题。在本工作中，我们基于“OOD数据应具有比训练数据更大的梯度范数”这一直观假设，分析利用数据点相对于深度生成模型参数的梯度进行OOD检测。我们将梯度大小的度量形式化为对费舍尔信息度量的近似。我们证明费舍尔信息矩阵（FIM）具有较大的绝对对角值，这启发了使用卡方分布的逐层梯度范数作为特征。我们将这些特征组合成一种简单、模型无关且无需超参数调优的OOD检测方法，该方法通过估计给定数据点的逐层梯度范数联合密度实现。我们发现这些逐层梯度范数呈弱相关性，使得其组合使用具有信息价值，并证明逐层梯度范数满足（数据表示）不变性原理。实证结果表明，该方法在大多数深度生成模型与图像数据集配对中优于典型性检验。