Feature engineering remains a critical yet challenging bottleneck in machine learning, particularly for tabular data, as identifying optimal features from an exponentially large feature space traditionally demands substantial domain expertise. To address this challenge, we introduce FAMOSE (Feature AugMentation and Optimal Selection agEnt), a novel framework that leverages the ReAct paradigm to autonomously explore, generate, and refine features while integrating feature selection and evaluation tools within an agent architecture. To our knowledge, FAMOSE represents the first application of an agentic ReAct framework to automated feature engineering, especially for both regression and classification tasks. Extensive experiments demonstrate that FAMOSE is at or near the state-of-the-art on classification tasks (especially tasks with more than 10K instances, where ROC-AUC increases 0.23% on average), and achieves the state-of-the-art for regression tasks by reducing RMSE by 2.0% on average, while remaining more robust to errors than other algorithms. We hypothesize that FAMOSE's strong performance is because ReAct allows the LLM context window to record (via iterative feature discovery and evaluation steps) what features did or did not work. This is similar to a few-shot prompt and guides the LLM to invent better, more innovative features. Our work offers evidence that AI agents are remarkably effective in solving problems that require highly inventive solutions, such as feature engineering.

翻译：特征工程仍然是机器学习中关键而具有挑战性的瓶颈，尤其是在表格数据领域，因为从指数级庞大的特征空间中识别最优特征传统上需要大量的领域专业知识。为应对这一挑战，我们提出了FAMOSE（特征增强与最优选择智能体），这是一个新颖的框架，它利用ReAct范式自主探索、生成和优化特征，同时将特征选择与评估工具集成于智能体架构之中。据我们所知，FAMOSE是首个将智能体ReAct框架应用于自动化特征工程的研究，尤其针对回归和分类任务。大量实验表明，FAMOSE在分类任务上达到或接近最先进水平（特别是在实例数超过10K的任务中，ROC-AUC平均提升0.23%），并在回归任务上通过平均降低2.0%的RMSE实现了最优性能，同时相比其他算法具有更强的错误鲁棒性。我们假设FAMOSE的优异性能源于ReAct机制允许大语言模型上下文窗口通过迭代的特征发现与评估步骤记录哪些特征有效或无效。这类似于少样本提示，能够引导大语言模型创造更优、更具创新性的特征。我们的工作证明，人工智能智能体在解决需要高度创造性解决方案的问题（如特征工程）方面具有显著效力。