Complexity is a fundamental concept underlying statistical learning theory that aims to inform generalization performance. Parameter count, while successful in low-dimensional settings, is not well-justified for overparameterized settings when the number of parameters is more than the number of training samples. We revisit complexity measures based on Rissanen's principle of minimum description length (MDL) and define a novel MDL-based complexity (MDL-COMP) that remains valid for overparameterized models. MDL-COMP is defined via an optimality criterion over the encodings induced by a good Ridge estimator class. We provide an extensive theoretical characterization of MDL-COMP for linear models and kernel methods and show that it is not just a function of parameter count, but rather a function of the singular values of the design or the kernel matrix and the signal-to-noise ratio. For a linear model with $n$ observations, $d$ parameters, and i.i.d. Gaussian predictors, MDL-COMP scales linearly with $d$ when $d<n$, but the scaling is exponentially smaller -- $\log d$ for $d>n$. For kernel methods, we show that MDL-COMP informs minimax in-sample error, and can decrease as the dimensionality of the input increases. We also prove that MDL-COMP upper bounds the in-sample mean squared error (MSE). Via an array of simulations and real-data experiments, we show that a data-driven Prac-MDL-COMP informs hyper-parameter tuning for optimizing test MSE with ridge regression in limited data settings, sometimes improving upon cross-validation and (always) saving computational costs. Finally, our findings also suggest that the recently observed double decent phenomenons in overparameterized models might be a consequence of the choice of non-ideal estimators.

翻译：复杂度是统计学习理论中的核心概念，旨在指导泛化性能。参数数量在低维场景中有效，但在过参数化场景（参数数量超过训练样本数）下缺乏合理性。我们基于Rissanen的最小描述长度（MDL）原则重新审视复杂度度量，并定义了一种新颖的MDL复杂度（MDL-COMP），使其在过参数化模型中依然有效。MDL-COMP通过最优性准则定义于由优质Ridge估计器类产生的编码之上。我们为线性模型和核方法提供了MDL-COMP的详尽理论刻画，表明它并非仅仅是参数数量的函数，而是设计矩阵或核矩阵的奇异值以及信噪比的函数。对于包含$n$个观测值、$d$个参数且独立同分布高斯预测变量的线性模型，当$d<n$时MDL-COMP与$d$呈线性增长，但当$d>n$时其缩放比例呈指数级缩小——仅为$\log d$。对于核方法，我们证明MDL-COMP可指示极小极大样本内误差，并可能随输入维度的增加而降低。我们还证明了MDL-COMP是样本内均方误差（MSE）的上界。通过一系列模拟和真实数据实验，我们展示了数据驱动的Prac-MDL-COMP能够指导小数据场景下岭回归测试MSE的超参数调优，有时优于交叉验证，且始终能节省计算成本。最后，我们的研究还表明，近期观察到的过参数化模型中的双重下降现象可能是由于非理想估计器的选择所致。