We study statistical estimation in a student--teacher setting, where predictions from a pre-trained teacher are used to guide a student model. A standard approach is to train the student to directly match the teacher's outputs, which we refer to as student soft matching (SM). This approach directly propagates any systematic bias or mis-specification present in the teacher, thereby degrading the student's predictions. We propose and analyze an alternative scheme, known as residual-as-teacher (RaT), in which the teacher is used to estimate residuals in the student's predictions. Our analysis shows how the student can thereby emulate a proximal gradient scheme for solving an oracle optimization problem, and this provably reduces the effect of teacher bias. For general student--teacher pairs, we establish non-asymptotic excess risk bounds for any RaT fixed point, along with convergence guarantees for the student-teacher iterative scheme. For kernel-based student--teacher pairs, we prove a sharp separation: the RaT method achieves the minimax-optimal rate, while the SM method incurs constant prediction error for any sample size. Experiments on both synthetic data and ImageNette classification under covariate shift corroborate our theoretical findings.

翻译：我们研究师生设定下的统计估计问题，其中预训练教师的预测用于指导学生模型。标准方法是训练学生直接匹配教师输出，我们称之为学生软匹配（SM）。该方法会直接传播教师模型中存在的系统性偏差或设定错误，从而降低学生的预测质量。我们提出并分析了一种替代方案——残差为师（RaT），即利用教师来估计学生预测中的残差。我们的分析表明，学生可通过此方案模拟求解最优优化问题的近端梯度方法，从而可证明地减轻教师偏差的影响。对于一般师生对，我们建立了RaT任意不动点的非渐近过度风险界，同时给出了师生迭代方法的收敛性保证。对于基于核函数的师生对，我们证明了显著的分离性：RaT方法达到极小化最优速率，而SM方法在任何样本量下都会产生常数预测误差。在合成数据和协变量偏移下的ImageNette分类实验均验证了我们的理论结论。