As the field of automated machine learning (AutoML) advances, it becomes increasingly important to incorporate domain knowledge into these systems. We present an approach for doing so by harnessing the power of large language models (LLMs). Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset. The method produces both Python code for creating new features and explanations for the utility of the generated features. Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets - boosting mean ROC AUC performance from 0.798 to 0.822 across all dataset - similar to the improvement achieved by using a random forest instead of logistic regression on our datasets. Furthermore, CAAFE is interpretable by providing a textual explanation for each generated feature. CAAFE paves the way for more extensive semi-automation in data science tasks and emphasizes the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML. We release our $\href{https://github.com/automl/CAAFE}{code}$, a simple $\href{https://colab.research.google.com/drive/1mCA8xOAJZ4MaB_alZvyARTMjhl6RZf0a}{demo}$ and a $\href{https://pypi.org/project/caafe/}{python\ package}$.

翻译：随着自动化机器学习（AutoML）领域的进步，将领域知识纳入这些系统变得愈发重要。我们提出了一种利用大语言模型（LLMs）实现这一目标的方法。具体而言，我们引入了面向上下文的自动特征工程（CAAFE），这是一种针对表格数据的特征工程方法，它利用大语言模型基于数据集描述迭代地生成更多语义上有意义的特征。该方法同时生成创建新特征的Python代码，并解释所生成特征的效用。尽管方法简单，CAAFE在14个数据集中的11个上提升了性能——将整个数据集上的平均ROC AUC性能从0.798提升至0.822——这一提升幅度与在我们的数据集上使用随机森林替代逻辑回归所达到的效果相似。此外，CAAFE通过为每个生成的特征提供文本解释而具有可解释性。CAAFE为数据科学任务中更广泛的半自动化铺平了道路，并强调了上下文感知解决方案的重要性，这些解决方案可以将AutoML系统的范围扩展到语义AutoML。我们发布了我们的$\href{https://github.com/automl/CAAFE}{代码}$、一个简单的$\href{https://colab.research.google.com/drive/1mCA8xOAJZ4MaB_alZvyARTMjhl6RZf0a}{演示}$和一个$\href{https://pypi.org/project/caafe/}{Python包}$。