Interpreting the inner workings of deep learning models is crucial for establishing trust and ensuring model safety. Concept-based explanations have emerged as a superior approach that is more interpretable than feature attribution estimates such as pixel saliency. However, defining the concepts for the interpretability analysis biases the explanations by the user's expectations on the concepts. To address this, we propose a novel post-hoc unsupervised method that automatically uncovers the concepts learned by deep models during training. By decomposing the latent space of a layer in singular vectors and refining them by unsupervised clustering, we uncover concept vectors aligned with directions of high variance that are relevant to the model prediction, and that point to semantically distinct concepts. Our extensive experiments reveal that the majority of our concepts are readily understandable to humans, exhibit coherency, and bear relevance to the task at hand. Moreover, we showcase the practical utility of our method in dataset exploration, where our concept vectors successfully identify outlier training samples affected by various confounding factors. This novel exploration technique has remarkable versatility to data types and model architectures and it will facilitate the identification of biases and the discovery of sources of error within training data.

翻译：解释深度学习模型的内部运作机制对于建立信任和确保模型安全至关重要。基于概念的解释方法作为一种优于像素显著性等特征归因估计的方法出现，具有更高的可解释性。然而，在可解释性分析中定义概念会因用户对概念的期望而偏置解释结果。为解决这一问题，我们提出了一种新颖的事后无监督方法，该方法能够自动揭示深度模型在训练过程中学习到的概念。通过将某一层的潜在空间分解为奇异向量，并利用无监督聚类对这些向量进行精炼，我们发现了与模型预测相关的高方差方向对齐的概念向量，这些向量指向语义上不同的概念。大量实验表明，我们发现的大多数概念易于人类理解，具有一致性，并且与当前任务相关。此外，我们展示了该方法在数据集探索中的实际效用——我们的概念向量成功识别了受各种混杂因素影响的异常训练样本。这种新颖的探索技术对多种数据类型和模型架构具有显著的通用性，并将有助于识别训练数据中的偏差并发现错误来源。