We develop a general framework for analyzing representation costs of parametric data-fitting methods through their parameter-space regularizers. From this abstract perspective, we define representation costs for arbitrary parametric models and reveal their induced (native) function spaces. This unifies recent function-space views of data-fitting methods. We also prove that many natural results hold in this abstract setting, including representer theorems for parametric methods on their native spaces. The framework also rigorously connects parametric methods with their equivalent nonparametric descriptions under sufficient overparameterization. Classical methods and their native spaces, such as kernel methods / reproducing kernel Hilbert spaces, wavelets / Besov spaces, and shallow neural networks / variation spaces emerge as special cases of our abstract framework. A byproduct of "axiomatizing" the study of representation costs is that we also immediately obtain new results for deep neural networks: For depth-$L$ feedforward ReLU networks, their induced native spaces are $p$-normable quasi-Banach spaces with $p = 2/L$. This reveals that the inductive bias of deep neural networks (as given by the representation cost) cannot be captured by norms for depths $L > 2$.

翻译：我们建立了一个通用框架，通过参数空间正则化器来分析参数化数据拟合方法的表示代价。从这一抽象视角出发，我们定义了任意参数化模型的表示代价，并揭示了它们所诱导的（原生）函数空间，从而统一了近期关于数据拟合方法的函数空间观点。我们还证明了许多自然结果在该抽象框架下成立，包括参数方法在其原生空间上的表示定理。该框架还严格建立了参数化方法与其在充分过参数化条件下的等价非参数化描述之间的联系。经典方法及其原生空间，如核方法/再生核希尔伯特空间、小波/贝索夫空间以及浅层神经网络/变分空间，均作为我们抽象框架的特例出现。对表示代价研究进行“公理化”的一个副产品是，我们立即获得了深度神经网络的新结果：对于深度为$L$的前馈ReLU网络，其诱导的原生空间是$p = 2/L$的$p$-范数化拟巴拿赫空间。这揭示了深度神经网络（由表示代价给出的）归纳偏好无法在深度$L > 2$时用范数来刻画。