Trained models are often composed with post-hoc transforms such as temperature scaling (TS), ensembling and stochastic weight averaging (SWA) to improve performance, robustness, uncertainty estimation, etc. However, such transforms are typically applied only after the base models have already been finalized by standard means. In this paper, we challenge this practice with an extensive empirical study. In particular, we demonstrate a phenomenon that we call post-hoc reversal, where performance trends are reversed after applying these post-hoc transforms. This phenomenon is especially prominent in high-noise settings. For example, while base models overfit badly early in training, both conventional ensembling and SWA favor base models trained for more epochs. Post-hoc reversal can also suppress the appearance of double descent and mitigate mismatches between test loss and test error seen in base models. Based on our findings, we propose post-hoc selection, a simple technique whereby post-hoc metrics inform model development decisions such as early stopping, checkpointing, and broader hyperparameter choices. Our experimental analyses span real-world vision, language, tabular and graph datasets from domains like satellite imaging, language modeling, census prediction and social network analysis. On an LLM instruction tuning dataset, post-hoc selection results in > 1.5x MMLU improvement compared to naive selection. Code is available at https://github.com/rishabh-ranjan/post-hoc-reversal.

翻译：训练后的模型通常通过事后变换（如温度缩放、集成学习和随机权重平均）进行组合，以提升性能、鲁棒性、不确定性估计等。然而，这些变换通常仅在基础模型已通过标准方法确定后才被应用。本文通过广泛的实证研究挑战了这一实践。具体而言，我们揭示了一种称为“事后反转”的现象，即应用这些事后变换后，性能趋势会发生逆转。这一现象在高噪声场景中尤为显著。例如，尽管基础模型在训练早期严重过拟合，但传统的集成学习和随机权重平均却更倾向于经过更多轮训练的基础模型。事后反转还可抑制双重下降的出现，并缓解基础模型中测试损失与测试错误之间的不匹配。基于我们的发现，我们提出了“事后选择”这一简单技术，即利用事后指标指导模型开发决策，如早停、检查点选择及更广泛的超参数选择。我们的实验分析涵盖了来自卫星成像、语言建模、人口普查预测和社会网络分析等领域的真实视觉、语言、表格和图数据集。在LLM指令微调数据集上，事后选择相比朴素选择实现了超过1.5倍的MMLU提升。代码见 https://github.com/rishabh-ranjan/post-hoc-reversal。