With infinitely many high-quality data points, infinite computational power, an infinitely large foundation model with a perfect training algorithm and guaranteed zero generalization error on the pretext task, can the model be used for everything? This question cannot be answered by the existing theory of representation, optimization or generalization, because the issues they mainly investigate are assumed to be nonexistent here. In this paper, we show that category theory provides powerful machinery to answer this question. We have proved three results. The first one limits the power of prompt-based learning, saying that the model can solve a downstream task with prompts if and only if the task is representable. The second one says fine tuning does not have this limit, as a foundation model with the minimum required power (up to symmetry) can theoretically solve downstream tasks for the category defined by pretext task, with fine tuning and enough resources. Our final result can be seen as a new type of generalization theorem, showing that the foundation model can generate unseen objects from the target category (e.g., images) using the structural information from the source category (e.g., texts). Along the way, we provide a categorical framework for supervised and self-supervised learning, which might be of independent interest.

翻译：假设拥有无限数量的高质量数据点、无限的计算能力、一个具有完美训练算法且在预文本任务上保证零泛化误差的无限大规模基础模型，该模型是否可用于解决一切问题？现有关于表征、优化或泛化的理论无法回答这一问题，因为这些理论所主要探讨的问题在此设定下被视为不存在。本文证明，范畴论为回答这一问题提供了强有力的工具。我们证明了三个结果：第一个结果限制了基于提示学习的效力，表明模型仅当下游任务具有可表示性时，才能通过提示解决该任务。第二个结果指出微调不存在这一限制——具备最低必要能力（对称性意义下）的基础模型，理论上可通过微调和充足资源，解决由预文本任务定义范畴内的任意下游任务。最后一个结果可被视为一类新型泛化定理，表明基础模型能利用源范畴（如文本）的结构信息，生成目标范畴（如图像）中未见过的对象。在此过程中，我们为监督学习和自监督学习提供了一个范畴论框架，该框架可能具有独立的研究价值。