Deep neural networks (DNNs), the agents of deep learning (DL), require a massive number of parallel/sequential operations. This makes it difficult to comprehend DNNs' operations and impedes proper diagnosis. Without better knowledge of their internal process, deploying DNNs in high-stakes domains can lead to catastrophic failures. Therefore, to build more reliable DNNs/DL to be deployed in high-stakes real-world problems, it is imperative that we gain insights into DNNs' internal operations underlying their decision-making. Here, we use the self-organizing map (SOM) to analyze DL models' internal codes associated with DNNs' decision-making. Our analyses suggest that shallow layers close to the input layer compress features into condensed space and that deep layers close to the output layer expand feature space. We also found evidence indicating that compressed features may underlie DNNs' vulnerabilities to adversarial perturbations.

翻译：深度神经网络（DNN）作为深度学习（DL）的载体，需要执行大量并行/串行运算。这使得理解DNN的运行机制变得困难，并阻碍了对其进行有效诊断。若缺乏对其内部过程的深入认识，在高风险领域部署DNN可能导致灾难性故障。因此，为使更可靠的DNN/DL能够应用于高风险现实问题，我们必须深入理解DNN决策过程背后的内部运行机制。本文采用自组织映射（SOM）方法分析DL模型与DNN决策相关的内部编码。分析表明，靠近输入层的浅层网络将特征压缩至紧凑空间，而靠近输出层的深层网络则扩展特征空间。我们还发现证据表明，压缩特征可能是导致DNN易受对抗扰动攻击的根本原因。