Meta-learning has emerged as a powerful approach to train neural networks to learn new tasks quickly from limited data. Broad exposure to different tasks leads to versatile representations enabling general problem solving. But, what are the limits of meta-learning? In this work, we explore the potential of amortizing the most powerful universal predictor, namely Solomonoff Induction (SI), into neural networks via leveraging meta-learning to its limits. We use Universal Turing Machines (UTMs) to generate training data used to expose networks to a broad range of patterns. We provide theoretical analysis of the UTM data generation processes and meta-training protocols. We conduct comprehensive experiments with neural architectures (e.g. LSTMs, Transformers) and algorithmic data generators of varying complexity and universality. Our results suggest that UTM data is a valuable resource for meta-learning, and that it can be used to train neural networks capable of learning universal prediction strategies.

翻译：元学习已成为一种强大的方法，用于训练神经网络从有限数据中快速学习新任务。广泛接触不同任务会产生多功能的表征，从而实现通用问题解决。但元学习的极限是什么？在这项工作中，我们通过将元学习发挥到极致，探索将最强大的通用预测器——即所罗门诺夫归纳法——摊销到神经网络中的潜力。我们使用通用图灵机生成训练数据，使网络暴露于广泛的模式中。我们提供了UTM数据生成过程和元训练协议的理论分析。我们对不同复杂性和通用性的神经架构（如LSTM、Transformer）和算法数据生成器进行了全面实验。我们的结果表明，UTM数据是元学习的宝贵资源，可用于训练能够学习通用预测策略的神经网络。