When upgrading neural models to a newer version, new errors that were not encountered in the legacy version can be introduced, known as regression errors. This inconsistent behavior during model upgrade often outweighs the benefits of accuracy gain and hinders the adoption of new models. To mitigate regression errors from model upgrade, distillation and ensemble have proven to be viable solutions without significant compromise in performance. Despite the progress, these approaches attained an incremental reduction in regression which is still far from achieving backward-compatible model upgrade. In this work, we propose a novel method, Gated Fusion, that promotes backward compatibility via learning to mix predictions between old and new models. Empirical results on two distinct model upgrade scenarios show that our method reduces the number of regression errors by 62% on average, outperforming the strongest baseline by an average of 25%.

翻译：当将神经模型升级至新版本时，可能引入遗留版本中未曾出现的新错误，即回归误差。模型升级过程中的这种不一致行为常超过准确率提升所带来的收益，阻碍了新模型的采用。为缓解模型升级导致的回归误差，蒸馏和集成已被证明是可行的解决方案，且不会显著牺牲性能。尽管取得进展，这些方法在减少回归方面仅实现渐进式改进，仍远未达到向后兼容的模型升级目标。本文提出一种新方法——门控融合（Gated Fusion），通过学习混合新旧模型预测结果来促进向后兼容性。在两种不同模型升级场景下的实验结果表明，该方法平均减少了62%的回归误差，比最强基线平均高出25%。