Deep neural networks give us a powerful method to model the training dataset's relationship between input and output. We can regard that as a complex adaptive system consisting of many artificial neurons that work as an adaptive memory as a whole. The network's behavior is training dynamics with a feedback loop from the evaluation of the loss function. We already know the training response can be constant or shows power law-like aging in some ideal situations. However, we still have gaps between those findings and other complex phenomena, like network fragility. To fill the gap, we introduce a very simple network and analyze it. We show the training response consists of some different factors based on training stages, activation functions, or training methods. In addition, we show feature space reduction as an effect of stochastic training dynamics, which can result in network fragility. Finally, we discuss some complex phenomena of deep networks.

翻译：深度神经网络为我们提供了一种强大的方法，用于建模训练数据集中输入与输出之间的关系。我们可以将其视为一个由众多人工神经元组成的复杂自适应系统，这些神经元作为一个整体发挥着自适应记忆的作用。网络的行为是一种具有从损失函数评估反馈回路的训练动态过程。我们已经知道，在某些理想情况下，训练响应可以是恒定的，或表现出类似幂律的老化现象。然而，这些发现与其他复杂现象（如网络脆弱性）之间仍存在差距。为弥补这一差距，我们引入了一个非常简单的网络并对其进行分析。我们表明，训练响应由基于训练阶段、激活函数或训练方法的不同因素组成。此外，我们还展示了特征空间缩减作为随机训练动态的一种效应，这可能导致网络脆弱性。最后，我们讨论了深度网络的一些复杂现象。