Pre-training has achieved remarkable success when transferred to downstream tasks. In machine learning, we care about not only the good performance of a model but also its behavior under reasonable shifts of condition. The same philosophy holds when pre-training a foundation model. However, the foundation model may not uniformly behave well for a series of related downstream tasks. This happens, for example, when conducting mask recovery regression where the recovery ability or the training instances diverge like pattern features are extracted dominantly on pre-training, but semantic features are also required on a downstream task. This paper considers pre-training a model that guarantees a uniformly good performance over the downstream tasks. We call this goal as $\textit{downstream-task robustness}$. Our method first separates the upstream task into several representative ones and applies a simple minimax loss for pre-training. We then design an efficient algorithm to solve the minimax loss and prove its convergence in the convex setting. In the experiments, we show both on large-scale natural language processing and computer vision datasets our method increases the metrics on worse-case downstream tasks. Additionally, some theoretical explanations for why our loss is beneficial are provided. Specifically, we show fewer samples are inherently required for the most challenging downstream task in some cases.

翻译：预训练在迁移至下游任务时已取得显著成功。在机器学习中，我们不仅关注模型的优异性能，更关注其在条件合理变化下的行为表现。这一理念同样适用于基础模型的预训练阶段。然而，当面对一系列相关的下游任务时，基础模型可能无法在所有任务中保持均匀的良好表现。例如在掩码恢复回归任务中，当恢复能力或训练实例存在差异——预训练阶段主要提取模式特征，但下游任务同时需要语义特征时，便会发生这种情况。本文旨在研究如何预训练一个能保证在下游任务中具有均匀优良性能的模型，我们将此目标称为"下游任务鲁棒性"。我们的方法首先将上游任务分解为若干代表性子任务，并采用简洁的极小极大损失函数进行预训练。随后我们设计了一种高效算法来求解该极小极大损失，并在凸优化设定下证明其收敛性。实验表明，在大型自然语言处理和计算机视觉数据集上，我们的方法提升了最差场景下游任务的评价指标。此外，我们还提供了理论解释说明该损失函数的优势：在某些情况下，最具挑战性的下游任务本质上需要的样本数量更少。