It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models. The phenomenon of weak-to-strong generalization exemplifies the advantage of this two-stage procedure: a strong student is trained on imperfect labels obtained from a weak teacher, and yet the strong student outperforms the weak teacher. In this paper, we show that the potential improvement is substantial, in the sense that it affects the scaling law followed by the test error. Specifically, we consider students and teachers trained via random feature ridge regression (RFRR). Our main technical contribution is to derive a deterministic equivalent for the excess test error of the student trained on labels obtained via the teacher. Via this deterministic equivalent, we then identify regimes in which the scaling law of the student improves upon that of the teacher, unveiling that the improvement can be achieved both in bias-dominated and variance-dominated settings. Strikingly, the student may attain the minimax optimal rate regardless of the scaling law of the teacher -- in fact, when the test error of the teacher does not even decay with the sample size.

翻译：在机器学习中，使用学习模型来标注数据，然后利用这些数据训练更强大的模型的做法日益普遍。弱到强泛化现象体现了这种两阶段流程的优势：一个强大的学生模型基于从弱教师模型获得的不完美标签进行训练，但其表现却优于弱教师模型。在本文中，我们表明这种潜在改进是显著的，因为它影响了测试误差遵循的缩放律。具体而言，我们考虑通过随机特征岭回归（RFRR）训练的学生和教师模型。我们的主要技术贡献是推导出学生模型在基于教师模型获得的标签上训练时的超额测试误差的确定性等价形式。通过这一确定性等价形式，我们识别出学生模型的缩放律相对于教师模型有所改进的区间，揭示了这种改进既可以在偏差主导的场景中实现，也可以在方差主导的场景中实现。令人瞩目的是，无论教师模型的缩放律如何，学生模型都可能达到极小化最优速率——实际上，即使教师模型的测试误差不随样本量增加而衰减，这一结论依然成立。