Neural Collapse is a phenomenon that helps identify sparse and low rank structures in deep classifiers. Recent work has extended the definition of neural collapse to regression problems, albeit only measuring the phenomenon at the last layer. In this paper, we establish that Neural Regression Collapse (NRC) also occurs below the last layer across different types of models. We show that in the collapsed layers of neural regression models, features lie in a subspace that corresponds to the target dimension, the feature covariance aligns with the target covariance, the input subspace of the layer weights aligns with the feature subspace, and the linear prediction error of the features is close to the overall prediction error of the model. In addition to establishing Deep NRC, we also show that models that exhibit Deep NRC learn the intrinsic dimension of low rank targets and explore the necessity of weight decay in inducing Deep NRC. This paper provides a more complete picture of the simple structure learned by deep networks in the context of regression.

翻译：神经坍缩是一种有助于识别深度分类器中稀疏低秩结构的现象。近期研究将神经坍缩的定义扩展至回归问题，但仅限于测量最后一层的现象。本文证实神经回归坍缩（NRC）同样出现在不同类型模型的浅层。研究表明，在神经回归模型的坍缩层中，特征位于与目标维度对应的子空间内，特征协方差与目标协方差对齐，层权重的输入子空间与特征子空间一致，且特征的线性预测误差接近模型的整体预测误差。除确立深度NRC外，我们还发现呈现深度NRC的模型能学习低秩目标的内在维度，并探究了权重衰减对诱导深度NRC的必要性。本文为深度网络在回归背景下所学习的简单结构提供了更完整的图景。