Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized & unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 & GPT-like models across multiple diverse setups. By scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised finetuning based NLP tasks. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization. On 0-shot MMLU, UL2 20B outperforms T0 and T5 models. UL2 20B also works well with chain-of-thought prompting and reasoning, making it an appealing choice for research into reasoning at a small to medium scale of 20B parameters. Finally, we apply FLAN instruction tuning to the UL2 20B model, achieving MMLU and Big-Bench scores competitive to FLAN-PaLM 62B. We release Flax-based T5X checkpoints for the UL2 20B & Flan-UL2 20B.

翻译：现有预训练模型通常面向特定类型的问题。至今，关于何种架构和预训练设置最为合适仍缺乏共识。本文提出一种统一框架，用于预训练在各类数据集和设置中普遍有效的模型。我们首先将架构原型与预训练目标——这两个常被混淆的概念——进行解耦。接着，我们提出自然语言处理中自监督学习的广义统一视角，展示不同预训练目标如何相互转化，以及目标间的插值如何有效。随后，我们提出混合去噪器（Mixture-of-Denoisers，MoD），这是一种融合多种预训练范式的预训练目标。我们还引入模式切换概念，使下游微调与特定预训练方案相关联。我们开展大量消融实验对比多种预训练目标，发现我们的方法通过超越T5和GPT类模型在多种不同设置下的表现，推动了帕累托前沿。通过将模型扩展至200亿参数，我们在50个成熟的监督微调NLP任务上达到最先进性能。我们的模型在上下文学习中同样表现强劲，零样本SuperGLUE超越1750亿参数的GPT-3，单样本摘要任务性能达到T5-XXL的三倍。在零样本MMLU上，UL2 20B优于T0和T5模型。UL2 20B在思维链提示与推理任务中表现优异，使其成为200亿参数规模推理研究的理想选择。最终，我们将FLAN指令微调应用于UL2 20B模型，在MMLU和Big-Bench上获得与FLAN-PaLM 62B相当的分数。我们开源了基于Flax的T5X检查点，包括UL2 20B及Flan-UL2 20B。