Abstraction is the process of extracting the essential features from raw data while ignoring irrelevant details. It is well known that abstraction emerges with depth in neural networks, where deep layers capture abstract characteristics of data by combining lower level features encoded in shallow layers (e.g. edges). Yet we argue that depth alone is not enough to develop truly abstract representations. We advocate that the level of abstraction crucially depends on how broad the training set is. We address the issue within a renormalisation group approach where a representation is expanded to encompass a broader set of data. We take the unique fixed point of this transformation -- the Hierarchical Feature Model -- as a candidate for a representation which is absolutely abstract. This theoretical picture is tested in numerical experiments based on Deep Belief Networks and auto-encoders trained on data of different breadth. These show that representations in neural networks approach the Hierarchical Feature Model as the data get broader and as depth increases, in agreement with theoretical predictions.

翻译：抽象化是从原始数据中提取本质特征而忽略无关细节的过程。众所周知，抽象化随着神经网络深度的增加而出现，深层通过组合浅层编码的低级特征（如边缘）来捕获数据的抽象特征。然而我们认为，仅凭深度不足以发展出真正抽象的表征。我们主张抽象化水平关键取决于训练集的广度。我们在重整化群方法的框架下探讨这一问题，其中表征被扩展以涵盖更广泛的数据集。我们将此变换的唯一不动点——层次特征模型——视为绝对抽象表征的候选者。这一理论图景在基于深度信念网络和自编码器的数值实验中得到了验证，这些网络在不同广度的数据上进行训练。实验表明，随着数据广度的增加和网络深度的加深，神经网络中的表征逐渐趋近层次特征模型，这与理论预测一致。