This paper proposes a scenario-based functional testing approach for enhancing the performance of machine learning (ML) applications. The proposed method is an iterative process that starts with testing the ML model on various scenarios to identify areas of weakness. It follows by a further testing on the suspected weak scenarios and statistically evaluate the model's performance on the scenarios to confirm the diagnosis. Once the diagnosis of weak scenarios is confirmed by test results, the treatment of the model is performed by retraining the model using a transfer learning technique with the original model as the base and applying a set of training data specifically targeting the treated scenarios plus a subset of training data selected at random from the original train dataset to prevent the so-call catastrophic forgetting effect. Finally, after the treatment, the model is assessed and evaluated again by testing on the treated scenarios as well as other scenarios to check if the treatment is effective and no side effect caused. The paper reports a case study with a real ML deep neural network (DNN) model, which is the perception system of an autonomous racing car. It is demonstrated that the method is effective in the sense that DNN model's performance can be improved. It provides an efficient method of enhancing ML model's performance with much less human and compute resource than retrain from scratch.

翻译：本文提出了一种基于场景的功能测试方法，用于提升机器学习（ML）应用的性能。该方法是一个迭代过程：首先在各种场景下测试ML模型以识别薄弱环节，随后针对疑似薄弱场景进行深入测试，并基于测试结果对模型在这些场景下的表现进行统计评估以确认诊断。一旦薄弱场景的诊断得到测试结果的验证，则通过迁移学习技术对模型进行治疗——以原始模型为基础，使用一组专门针对治疗场景的训练数据，并结合从原始训练数据集中随机选取的部分数据（以防止所谓的灾难性遗忘效应）进行重新训练。最后，在治疗后，需再次对模型进行评估，测试其在治疗场景及其他场景下的表现，以确认治疗效果及未引发副作用。本文基于一个真实ML深度神经网络（DNN）模型（即自动驾驶赛车的感知系统）开展了案例研究，结果表明该方法能够有效提升DNN模型性能。与从头开始重新训练相比，该方法提供了一种高效提升ML模型性能的途径，且所需的人力和计算资源显著减少。