How the depth and width of neural networks shape learned representations is difficult to predict from the data and architecture. We use neural collapse (NC) as a probe of the representation geometry and express its first three properties, within-class variability collapse (NC1), simplex class means (NC2) and classifier alignment (NC3), through the learned feature kernel, extending a known identity for NC1 to NC2 and NC3. For deep linear networks with an exchangeable input kernel, the kernel equations yield a geometric law, exact within this theory, for the layerwise decrease of NC1. At large width $N$ the accumulated decrease becomes a function of the depth-to-width ratio $L/N$, with its rate determined self-consistently by the data and architecture, yielding a scaling law for representation geometry.
翻译:暂无翻译