Pretraining is a popular and powerful paradigm in machine learning. As an example, suppose one has a modest-sized dataset of images of cats and dogs, and plans to fit a deep neural network to classify them from the pixel features. With pretraining, we start with a neural network trained on a large corpus of images, consisting of not just cats and dogs but hundreds of other image types. Then we fix all of the network weights except for the top layer (which makes the final classification) and train (or "fine tune") those weights on our dataset. This often results in dramatically better performance than the network trained solely on our smaller dataset. In this paper, we ask the question "Can pretraining help the lasso?". We develop a framework for the lasso in which an overall model is fit to a large set of data, and then fine-tuned to a specific task on a smaller dataset. This latter dataset can be a subset of the original dataset, but does not need to be. We find that this framework has a wide variety of applications, including stratified models, multinomial targets, multi-response models, conditional average treatment estimation and even gradient boosting. In the stratified model setting, the pretrained lasso pipeline estimates the coefficients common to all groups at the first stage, and then group specific coefficients at the second "fine-tuning" stage. We show that under appropriate assumptions, the support recovery rate of the common coefficients is superior to that of the usual lasso trained only on individual groups. This separate identification of common and individual coefficients can also be useful for scientific understanding.

翻译：预训练是机器学习中一种流行且强大的范式。例如，假设我们有一个中等大小的猫狗图像数据集，并计划拟合深度神经网络以根据像素特征对其进行分类。通过预训练，我们首先使用一个在大型图像语料库（不仅包含猫狗，还包括数百种其他图像类型）上训练的神经网络。然后，除顶层（负责最终分类）外，固定所有网络权重，并在我们的数据集上训练（或“微调”）这些权重。这通常比仅使用较小数据集训练的网络表现出显著更优的性能。在本文中，我们提出一个问题：“预训练能否帮助Lasso？”我们为Lasso开发了一个框架，其中先对整个大规模数据集拟合一个全局模型，然后在较小数据集上针对特定任务进行微调。后者可以是原始数据集的子集，但非必须。我们发现该框架具有广泛的应用，包括分层模型、多项目标、多响应模型、条件平均处理效应估计甚至梯度提升。在分层模型设置中，预训练Lasso流程在第一阶段估计所有组共有的系数，然后在第二阶段（“微调”阶段）估计各组特有的系数。我们证明，在适当假设下，共有系数的支持恢复率优于仅针对单个组训练的常规Lasso。这种对共有系数和个体系数的分离识别也有助于科学理解。