Deep learning algorithms demonstrate a surprising ability to learn high-dimensional tasks from limited examples. This is commonly attributed to the depth of neural networks, enabling them to build a hierarchy of abstract, low-dimensional data representations. However, how many training examples are required to learn such representations remains unknown. To quantitatively study this question, we introduce the Random Hierarchy Model: a family of synthetic tasks inspired by the hierarchical structure of language and images. The model is a classification task where each class corresponds to a group of high-level features, chosen among several equivalent groups associated with the same class. In turn, each feature corresponds to a group of sub-features chosen among several equivalent ones and so on, following a hierarchy of composition rules. We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups. Moreover, the number of data required corresponds to the point where correlations between low-level features and classes become detectable. Overall, our results indicate how deep networks overcome the curse of dimensionality by building invariant representations, and provide an estimate of the number of data required to learn a hierarchical task.

翻译：深度学习算法展现出从有限样本中学习高维任务的惊人能力。这一现象通常归因于神经网络的深度，使其能够构建抽象、低维数据表示的层级结构。然而，学习此类表示所需训练样本的数量仍属未知。为定量研究该问题，我们引入随机层级模型：一系列受语言与图像层级结构启发的合成任务。该模型是一个分类任务，其中每个类别对应一组高级特征，这些特征从与同一类别关联的若干等效组中选取。相应地，每个特征对应一组子特征，这些子特征从若干等效子特征组中选取，并依循组合规则的层级结构。我们发现深度网络通过学习对交换等效组具有不变性的内部表示来完成任务。此外，所需数据量对应于低层特征与类别之间相关性变得可检测的临界点。总体而言，我们的结果表明深度网络如何通过构建不变性表示克服维度灾难，并提供了学习层级任务所需数据量的估计。