Automated Machine Learning (AutoML) approaches encompass traditional methods that optimize fixed pipelines for model selection and ensembling, as well as newer LLM-based frameworks that autonomously build pipelines. While LLM-based agents have shown promise in automating machine learning tasks, they often generate low-diversity and suboptimal code, even after multiple iterations. To overcome these limitations, we introduce Tree-Search Enhanced LLM Agents (SELA), an innovative agent-based system that leverages Monte Carlo Tree Search (MCTS) to optimize the AutoML process. By representing pipeline configurations as trees, our framework enables agents to conduct experiments intelligently and iteratively refine their strategies, facilitating a more effective exploration of the machine learning solution space. This novel approach allows SELA to discover optimal pathways based on experimental feedback, improving the overall quality of the solutions. In an extensive evaluation across 20 machine learning datasets, we compare the performance of traditional and agent-based AutoML methods, demonstrating that SELA achieves a win rate of 65% to 80% against each baseline across all datasets. These results underscore the significant potential of agent-based strategies in AutoML, offering a fresh perspective on tackling complex machine learning challenges.

翻译：自动化机器学习（AutoML）方法既包括优化固定流程以进行模型选择与集成的传统方法，也包含能够自主构建流程的新型基于大语言模型（LLM）的框架。尽管基于LLM的智能体在自动化机器学习任务中展现出潜力，但它们即使在多次迭代后仍常生成多样性不足且次优的代码。为克服这些限制，我们提出了基于树搜索增强的大语言模型智能体（SELA），这是一个创新的基于智能体的系统，利用蒙特卡洛树搜索（MCTS）来优化AutoML流程。通过将流程配置表示为树状结构，我们的框架使智能体能够智能地进行实验并迭代优化其策略，从而更有效地探索机器学习解空间。这种新颖的方法使SELA能够基于实验反馈发现最优路径，提升解决方案的整体质量。在涵盖20个机器学习数据集的广泛评估中，我们比较了传统方法与基于智能体的AutoML方法的性能，结果表明SELA在所有数据集上相对于各基线方法均取得了65%至80%的胜率。这些结果凸显了基于智能体的策略在AutoML中的巨大潜力，为应对复杂的机器学习挑战提供了新的视角。