No free lunch theorems for supervised learning state that no learner can solve all problems or that all learners achieve exactly the same accuracy on average over a uniform distribution on learning problems. Accordingly, these theorems are often referenced in support of the notion that individual problems require specially tailored inductive biases. While virtually all uniformly sampled datasets have high complexity, real-world problems disproportionately generate low-complexity data, and we argue that neural network models share this same preference, formalized using Kolmogorov complexity. Notably, we show that architectures designed for a particular domain, such as computer vision, can compress datasets on a variety of seemingly unrelated domains. Our experiments show that pre-trained and even randomly initialized language models prefer to generate low-complexity sequences. Whereas no free lunch theorems seemingly indicate that individual problems require specialized learners, we explain how tasks that often require human intervention such as picking an appropriately sized model when labeled data is scarce or plentiful can be automated into a single learning algorithm. These observations justify the trend in deep learning of unifying seemingly disparate problems with an increasingly small set of machine learning models.

翻译：无免费午餐定理指出，在监督学习中，没有一种学习器能解决所有问题，或者所有学习器在学习问题的均匀分布上平均准确率完全相同。因此，这些定理常被引用来支持“每个问题都需要专门定制的归纳偏好”这一观点。虽然几乎所有均匀采样的数据集都具有高复杂度，但现实世界的问题往往产生低复杂度数据，我们论证神经网络模型也共享这一偏好，并用柯尔莫哥洛夫复杂度形式化地表述了这一点。值得注意的是，我们表明，为特定领域（如计算机视觉）设计的架构能够在看似不相关的多种领域上压缩数据集。我们的实验显示，预训练甚至随机初始化的语言模型倾向于生成低复杂度序列。尽管无免费午餐定理似乎表明每个问题都需要专门的学习器，但我们解释了如何将通常需要人工干预的任务（例如在标记数据稀缺或充足时选择合适的模型规模）自动化到单一学习算法中。这些观察结果验证了深度学习领域中将看似不同的问题用越来越少的机器学习模型统一处理的趋势。