Feature discovery from complex unstructured data is fundamentally a reasoning problem: it requires identifying abstractions that are predictive of a target outcome while avoiding leakage, proxies, and post-outcome signals. With the introduction of ever-improving Large Language Models (LLMs), our method provides a structured method for addressing this challenge. LLMs are well suited for this task by being able to process large amounts of information, but unconstrained feature generation can lead to weak features. In this work, we study reasoning control in LLMs by inducing cognitive behaviors for improving feature discovery. We introduce CoFEE (Cognitive Feature Engineering Engine), a reasoning control framework that enforces cognitive behaviors in how the LLM reasons during feature discovery. From a machine learning perspective, these cognitive behaviors act as structured inductive biases over the space of candidate features generated by the model. These behaviors have been exploited with success in ML models, and include backward chaining from outcomes, subgoal decomposition, verification against observability and leakage criteria, and explicit backtracking of rejected reasoning paths. In a controlled comparison, we show that enforcing cognitive behaviors yields features with higher empirical predictability than those under unconstrained vanilla LLM prompts. CoFEE achieves an average Success Rate Score that is 15.2% higher than the vanilla approach, while generating 29% fewer features and reducing costs by 53.3%. Using held-out feature evaluation, we assess whether cognitively induced features generalize beyond the data used for discovery. Our results indicate that, in our evaluated setting, reasoning control is associated with improvements in quality and efficiency of LLM-based feature discovery.

翻译：从复杂非结构化数据中进行特征发现本质上是一个推理问题：它要求识别能够预测目标结果的抽象概念，同时避免信息泄露、代理变量及事后信号。随着大语言模型的持续进步，我们提出了一种应对该挑战的结构化方法。LLM因具备处理海量信息的能力而非常适合此任务，但无约束的特征生成会导致弱特征。本研究通过诱导认知行为来改进特征发现，重点探索LLM的推理控制机制。我们提出了CoFEE（认知特征工程引擎），这是一个通过强制LLM在特征发现过程中采用特定认知行为实现推理控制的框架。从机器学习角度看，这些认知行为为模型生成的候选特征空间提供了结构化归纳偏置。这些行为已在机器学习模型中得到成功应用，包括结果反向链推理、子目标分解、可观测性与泄漏准则验证，以及显式回溯被拒推理路径。在控制对比实验中，我们证明强制认知行为所产生的特征，其经验可预测性显著高于无约束的原始LLM提示。CoFEE在生成特征数量减少29%、成本降低53.3%的情况下，平均成功率得分比原始方法高出15.2%。通过留出特征评估，我们检验了认知诱导特征是否能在发现数据之外保持泛化能力。结果表明，在我们评估的场景下，推理控制与基于LLM的特征发现的质量和效率提升存在关联。