Deep Learning models have become an integrated component of modern software systems. In response to the challenge of model design, researchers proposed Automated Machine Learning (AutoML) systems, which automatically search for model architecture and hyperparameters for a given task. Like other software systems, existing AutoML systems suffer from bugs. We identify two common and severe bugs in AutoML, performance bug (i.e., searching for the desired model takes an unreasonably long time) and ineffective search bug (i.e., AutoML systems are not able to find an accurate enough model). After analyzing the workflow of AutoML, we observe that existing AutoML systems overlook potential opportunities in search space, search method, and search feedback, which results in performance and ineffective search bugs. Based on our analysis, we design and implement DREAM, an automatic debugging and repairing system for AutoML systems. It monitors the process of AutoML to collect detailed feedback and automatically repairs bugs by expanding search space and leveraging a feedback-driven search strategy. Our evaluation results show that DREAM can effectively and efficiently repair AutoML bugs.

翻译：深度学习模型已成为现代软件系统的核心组成部分。为应对模型设计挑战，研究人员提出了自动化机器学习（AutoML）系统，该系统可针对给定任务自动搜索模型架构与超参数。与其他软件系统类似，现有AutoML系统存在程序缺陷。我们识别出AutoML中两类常见且严重的缺陷：性能缺陷（即搜索期望模型耗时过长）与无效搜索缺陷（即AutoML系统无法找到足够精确的模型）。通过分析AutoML工作流，我们发现现有AutoML系统在搜索空间、搜索方法和搜索反馈中忽略了潜在机会，导致性能缺陷与无效搜索缺陷。基于此分析，我们设计并实现了DREAM——面向AutoML系统的自动调试与修复系统。该系统通过监控AutoML运行过程收集细粒度反馈，并采用扩展搜索空间与反馈驱动搜索策略的方式自动修复缺陷。评估结果表明，DREAM能够高效且有效地修复AutoML系统缺陷。