Complexity is a fundamental concept underlying statistical learning theory that aims to inform generalization performance. Parameter count, while successful in low-dimensional settings, is not well-justified for overparameterized settings when the number of parameters is more than the number of training samples. We revisit complexity measures based on Rissanen's principle of minimum description length (MDL) and define a novel MDL-based complexity (MDL-COMP) that remains valid for overparameterized models. MDL-COMP is defined via an optimality criterion over the encodings induced by a good Ridge estimator class. We provide an extensive theoretical characterization of MDL-COMP for linear models and kernel methods and show that it is not just a function of parameter count, but rather a function of the singular values of the design or the kernel matrix and the signal-to-noise ratio. For a linear model with $n$ observations, $d$ parameters, and i.i.d. Gaussian predictors, MDL-COMP scales linearly with $d$ when $d<n$, but the scaling is exponentially smaller -- $\log d$ for $d>n$. For kernel methods, we show that MDL-COMP informs minimax in-sample error, and can decrease as the dimensionality of the input increases. We also prove that MDL-COMP upper bounds the in-sample mean squared error (MSE). Via an array of simulations and real-data experiments, we show that a data-driven Prac-MDL-COMP informs hyper-parameter tuning for optimizing test MSE with ridge regression in limited data settings, sometimes improving upon cross-validation and (always) saving computational costs. Finally, our findings also suggest that the recently observed double decent phenomenons in overparameterized models might be a consequence of the choice of non-ideal estimators.

翻译：复杂度是统计学习理论中的基本概念，旨在指导泛化性能。参数数量在低维设置中表现良好，但在参数数量超过训练样本数量的过参数化场景下缺乏合理性。我们基于Rissanen的最小描述长度（MDL）原则重新审视复杂度度量，并定义了一种新的基于MDL的复杂度（MDL-COMP），该复杂度在过参数化模型中依然有效。MDL-COMP通过由良好岭估计量类所诱导编码的最优性准则定义。我们对线性模型和核方法的MDL-COMP进行了详尽的理论刻画，表明其并非仅是参数数量的函数，而是设计矩阵或核矩阵的奇异值及信噪比的函数。对于具有n个观测值、d个参数且预测变量服从独立同分布高斯分布的线性模型，当d<n时，MDL-COMP随d线性增长；但当d>n时，其以指数级更小的尺度增长（即log d）。对于核方法，我们证明MDL-COMP能够体现极小极大样本内误差，并可随输入维度增加而降低。我们还证明MDL-COMP是样本内均方误差（MSE）的上界。通过一系列仿真和真实数据实验，我们展示了数据驱动的Prac-MDL-COMP能够指导有限数据场景下岭回归测试MSE的超参数调优，其效果有时优于交叉验证，且总能节省计算成本。最后，我们的发现还表明，近期在过参数化模型中观察到的双重下降现象可能源于非理想估计量的选择。