As the field of automated machine learning (AutoML) advances, it becomes increasingly important to incorporate domain knowledge into these systems. We present an approach for doing so by harnessing the power of large language models (LLMs). Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset. The method produces both Python code for creating new features and explanations for the utility of the generated features. Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets - boosting mean ROC AUC performance from 0.798 to 0.822 across all dataset - similar to the improvement achieved by using a random forest instead of logistic regression on our datasets. Furthermore, CAAFE is interpretable by providing a textual explanation for each generated feature. CAAFE paves the way for more extensive semi-automation in data science tasks and emphasizes the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML. We release our $\href{https://github.com/automl/CAAFE}{code}$, a simple $\href{https://colab.research.google.com/drive/1mCA8xOAJZ4MaB_alZvyARTMjhl6RZf0a}{demo}$ and a $\href{https://pypi.org/project/caafe/}{python\ package}$.

翻译：随着自动化机器学习（AutoML）领域的发展，将领域知识融入这些系统变得日益重要。我们提出了一种利用大语言模型（LLMs）实现这一目标的方法。具体而言，我们引入了上下文感知的自动特征工程（CAAFE），这是一种针对表格数据集的特征工程方法，它基于数据集描述，利用大语言模型迭代生成额外的语义上有意义的特征。该方法既生成创建新特征的Python代码，也生成这些生成特征实用性的解释。尽管在方法论上较为简单，但CAAFE在14个数据集中的11个上提升了性能——将所有数据集的平均ROC AUC性能从0.798提升至0.822，这与在我们的数据集上用随机森林替代逻辑回归所获得的提升幅度相近。此外，CAAFE具有可解释性，为每个生成的特征提供文本解释。CAAFE为数据科学任务中更广泛的半自动化铺平了道路，并强调了上下文感知解决方案的重要性，这些解决方案可将AutoML系统的范围扩展至语义AutoML。我们发布了我们的$\href{https://github.com/automl/CAAFE}{代码}$、一个简单的$\href{https://colab.research.google.com/drive/1mCA8xOAJZ4MaB_alZvyARTMjhl6RZf0a}{演示}$和一个$\href{https://pypi.org/project/caafe/}{Python包}$。