Deep learning models are complex due to their size, structure, and inherent randomness in training procedures. Additional complexity arises from the selection of datasets and inductive biases. Addressing these challenges for explainability, Kim et al. (2018) introduced Concept Activation Vectors (CAVs), which aim to understand deep models' internal states in terms of human-aligned concepts. These concepts correspond to directions in latent space, identified using linear discriminants. Although this method was first applied to image classification, it was later adapted to other domains, including natural language processing. In this work, we attempt to apply the method to electroencephalogram (EEG) data for explainability in Kostas et al.'s BENDR (2021), a large-scale transformer model. A crucial part of this endeavor involves defining the explanatory concepts and selecting relevant datasets to ground concepts in the latent space. Our focus is on two mechanisms for EEG concept formation: the use of externally labeled EEG datasets, and the application of anatomically defined concepts. The former approach is a straightforward generalization of methods used in image classification, while the latter is novel and specific to EEG. We present evidence that both approaches to concept formation yield valuable insights into the representations learned by deep EEG models.

翻译：深度学习模型因其规模、结构以及训练过程中的固有随机性而具有复杂性。数据集的选取和归纳偏差进一步增加了这种复杂性。为解决这些可解释性挑战，Kim等人（2018）引入了概念激活向量（CAVs），旨在通过人类校准的概念来理解深度模型的内部状态。这些概念对应于潜在空间中通过线性判别分析确定的方向。尽管该方法最初应用于图像分类，但后来被推广至包括自然语言处理在内的其他领域。在本研究中，我们尝试将该方法应用于脑电图（EEG）数据，以解释Kostas等人（2021）提出的大规模变压器模型BENDR。此项工作的关键环节包括定义解释性概念，以及选择相关数据集将概念锚定于潜在空间。我们聚焦于两种EEG概念形成机制：利用外部标注的EEG数据集，以及应用解剖学定义的概念。前者是图像分类中常用方法的直接推广，而后者则是针对EEG数据的新颖方法。我们提供的证据表明，这两种概念形成方法均能对深度EEG模型所学到的表示产生有价值的见解。