As the field of automated machine learning (AutoML) advances, it becomes increasingly important to incorporate domain knowledge into these systems. We present an approach for doing so by harnessing the power of large language models (LLMs). Specifically, we introduce Context-Aware Automated Feature Engineering (CAAFE), a feature engineering method for tabular datasets that utilizes an LLM to iteratively generate additional semantically meaningful features for tabular datasets based on the description of the dataset. The method produces both Python code for creating new features and explanations for the utility of the generated features. Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets - boosting mean ROC AUC performance from 0.798 to 0.822 across all dataset - similar to the improvement achieved by using a random forest instead of logistic regression on our datasets. Furthermore, CAAFE is interpretable by providing a textual explanation for each generated feature. CAAFE paves the way for more extensive semi-automation in data science tasks and emphasizes the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML. We release our $\href{https://github.com/automl/CAAFE}{code}$, a simple $\href{https://colab.research.google.com/drive/1mCA8xOAJZ4MaB_alZvyARTMjhl6RZf0a}{demo}$ and a $\href{https://pypi.org/project/caafe/}{python\ package}$.

翻译：随着自动机器学习（AutoML）领域的不断发展，将领域知识融入这些系统变得日益重要。我们提出了一种利用大语言模型（LLMs）实现这一目标的方法。具体而言，我们引入了上下文感知的自动特征工程（CAAFE），这是一种针对表格数据集的特征工程方法，通过利用大语言模型基于数据集描述迭代生成额外具有语义意义的特征。该方法既生成用于创建新特征的Python代码，也生成关于所产生特征实用性的解释。尽管方法学上简单，CAAFE在14个数据集中的11个上提升了性能——将所有数据集的平均ROC AUC性能从0.798提升至0.822——这与在我们的数据集上用随机森林替代逻辑回归所取得的改进相近。此外，CAAFE通过为每个生成的特征提供文本解释而具有可解释性。CAAFE为数据科学任务中更广泛的半自动化铺平了道路，并强调了上下文感知解决方案的重要性，这些解决方案可将AutoML系统的范围扩展至语义AutoML。我们公开发布了代码、简易演示和Python包。