By now there is substantial evidence that deep learning models learn certain human-interpretable features as part of their internal representations of data. As having the right (or wrong) concepts is critical to trustworthy machine learning systems, it is natural to ask which inputs from the model's original training set were most important for learning a concept at a given layer. To answer this, we combine data attribution methods with methods for probing the concepts learned by a model. Training network and probe ensembles for two concept datasets on a range of network layers, we use the recently developed TRAK method for large-scale data attribution. We find some evidence for convergence, where removing the 10,000 top attributing images for a concept and retraining the model does not change the location of the concept in the network nor the probing sparsity of the concept. This suggests that rather than being highly dependent on a few specific examples, the features that inform the development of a concept are spread in a more diffuse manner across its exemplars, implying robustness in concept formation.

翻译：目前已有充分证据表明，深度学习模型在内部数据表征中会习得某些人类可理解的语义特征。由于正确（或错误）概念的获取对可信机器学习系统至关重要，我们自然需要探究：在模型原始训练集中，哪些输入对特定层习得某个概念最为关键？为解答这一问题，我们将数据归因方法与模型习得概念的探测技术相结合。通过在多个网络层上对两类概念数据集训练网络集成与探测集成，我们采用近期发展的TRAK方法进行大规模数据归因。研究发现存在收敛性证据：即使移除某概念的10,000张最高归因图像并重新训练模型，该概念在网络中的定位及其探测稀疏性均未发生改变。这表明，促进概念形成的特征并非高度依赖少数特定样本，而是以更分散的方式分布在各类样本中，从而体现了概念形成的鲁棒性。