As Deep Neural Networks (DNNs) are rapidly being adopted within large software systems, software developers are increasingly required to design, train, and deploy such models into the systems they develop. Consequently, testing and improving the robustness of these models have received a lot of attention lately. However, relatively little effort has been made to address the difficulties developers experience when designing and training such models: if the evaluation of a model shows poor performance after the initial training, what should the developer change? We survey and evaluate existing state-of-the-art techniques that can be used to repair model performance, using a benchmark of both real-world mistakes developers made while designing DNN models and artificial faulty models generated by mutating the model code. The empirical evaluation shows that random baseline is comparable with or sometimes outperforms existing state-of-the-art techniques. However, for larger and more complicated models, all repair techniques fail to find fixes. Our findings call for further research to develop more sophisticated techniques for Deep Learning repair.

翻译：随着深度神经网络在大型软件系统中的快速普及，软件开发者日益需要设计、训练并将其部署到所开发的系统中。因此，近期提升这些模型鲁棒性的测试与改进工作备受关注。然而，针对开发者在设计与训练此类模型时遇到的困难，相关解决方案仍相对匮乏：当模型评估显示初始训练效果不佳时，开发者应如何调整？我们调研并评估了现有可用于修复模型性能的前沿技术，以开发者设计DNN模型时实际出现的错误及通过模型代码变异生成的人工缺陷模型为基准进行测试。实证评估表明，随机基线方法在性能上可与现有最优技术相媲美，甚至在某些情况下更优。但对于规模更大、结构更复杂的模型，所有修复技术均无法有效解决问题。这一发现表明，亟需进一步研究以开发更复杂的深度学习修复技术。