Compression and generalization are fundamentally related through Solomonoff induction and the minimum description length principle (MDL), which predict that simpler models generalize better when data arises from low-complexity distributions. In this article, we combine insights from algorithmic information theory and techniques from neural network pruning to improve model generalization by identifying the most effective data compression method. Since exact MDL optimization is intractable, we cast it as $\ell_0$ regularized learning and explain why parameter sparsity provides an effective computable approximation of model description length. To identify the best practical approach, we systematically compare and refine complementary sparse optimization methods. In particular, we improve probabilistic pruning through a procedure that does not require Monte Carlo sampling and refine smooth $\ell_0$ approximations with a binary search routine that reduces hyperparameter complexity. Across convolutional networks and transformers evaluated on image and text datasets, our refined methods improve upon their predecessors, achieve substantial model compression with minimal accuracy loss, and yield short data description lengths. Finally, we use these methods in a controlled teacher-student setting to empirically verify the prediction of Solomonoff induction that compressed models learn more sample-efficiently and generalize better.

翻译：压缩与泛化通过所罗门诺夫归纳法和最小描述长度原则（MDL）存在根本性关联——当数据源自低复杂度分布时，更简练的模型泛化能力更优。本文融合算法信息论洞见与神经网络剪枝技术，通过识别最有效的数据压缩方法提升模型泛化性能。由于精确MDL优化难以实现，我们将其转化为ℓ₀正则化学习问题，并论证参数稀疏性为何能作为模型描述长度的有效可计算近似。为确定最佳实践方案，我们系统比较并优化了互补的稀疏优化方法：通过无需蒙特卡洛采样的流程改进概率剪枝，利用二分搜索算法优化平滑ℓ₀近似以降低超参数复杂度。在图像和文本数据集上评估的卷积网络与Transformer中，改进方法较原版表现更优，在保持最小精度损失的同时实现显著模型压缩，并产生较短数据描述长度。最终，我们在受控师生学习场景中应用这些方法，实证验证了所罗门诺夫归纳法的预测——压缩模型具有更高的样本效率和更强的泛化能力。