Training foundation models on extensive datasets and then finetuning them on specific tasks has emerged as the mainstream approach in artificial intelligence. However, the model robustness, which is a critical aspect for safety, is often optimized for each specific task rather than at the pretraining stage. In this paper, we propose a method for pretraining certifiably robust models that can be readily finetuned for adaptation to a particular task. A key challenge is dealing with the compromise between semantic learning and robustness. We address this with a simple yet highly effective strategy based on significantly broadening the pretraining data distribution, which is shown to greatly benefit finetuning for downstream tasks. Through pretraining on a mixture of clean and various noisy images, we find that surprisingly strong certified accuracy can be achieved even when finetuning on only clean images. Furthermore, this strategy requires just a single model to deal with various noise levels, thus substantially reducing computational costs in relation to previous works that employ multiple models. Despite using just one model, our method can still yield results that are on par with, or even superior to, existing multi-model methods.

翻译：在大量数据集上训练基础模型，然后针对特定任务进行微调，已成为人工智能的主流方法。然而，模型鲁棒性作为安全性的关键方面，通常针对每个具体任务进行优化，而并非在预训练阶段考虑。本文提出一种预训练可认证鲁棒模型的方法，使模型能够轻松微调以适配特定任务。一个关键挑战在于处理语义学习与鲁棒性之间的权衡。我们通过基于显著扩展预训练数据分布的简单而高效策略来解决这一问题，该策略被证明对下游任务的微调大有裨益。通过在干净图像与多种噪声图像的混合数据上进行预训练，我们发现即使在仅微调于干净图像的情况下，也能获得出乎意料的强认证准确率。此外，该策略仅需单个模型即可处理多种噪声水平，从而相较于以往采用多个模型的方法大幅降低了计算成本。尽管仅使用单一模型，我们的方法仍能取得与现有多模型方法相当甚至更优的结果。