Tabular data high-stakes critical decision-making in domains such as finance, healthcare, and scientific discovery. Yet, learning effectively from tabular data in few-shot settings, where labeled examples are scarce, remains a fundamental challenge. Traditional tree-based methods often falter in these regimes due to their reliance on statistical purity metrics, which become unstable and prone to overfitting with limited supervision. At the same time, direct applications of large language models (LLMs) often overlook its inherent structure, leading to suboptimal performance. To overcome these limitations, we propose FORESTLLM, a novel framework that unifies the structural inductive biases of decision forests with the semantic reasoning capabilities of LLMs. Crucially, FORESTLLM leverages the LLM only during training, treating it as an offline model designer that encodes rich, contextual knowledge into a lightweight, interpretable forest model, eliminating the need for LLM inference at test time. Our method is two-fold. First, we introduce a semantic splitting criterion in which the LLM evaluates candidate partitions based on their coherence over both labeled and unlabeled data, enabling the induction of more robust and generalizable tree structures under few-shot supervision. Second, we propose a one-time in-context inference mechanism for leaf node stabilization, where the LLM distills the decision path and its supporting examples into a concise, deterministic prediction, replacing noisy empirical estimates with semantically informed outputs. Across a diverse suite of few-shot classification and regression benchmarks, FORESTLLM achieves state-of-the-art performance.

翻译：表格数据在金融、医疗和科学发现等领域的高风险关键决策中至关重要。然而，在小样本场景下，即标注示例稀缺时，如何从表格数据中有效学习仍然是一个根本性挑战。传统的基于树的方法在这些情况下常常表现不佳，因为它们依赖于统计纯度指标，而这些指标在有限监督下会变得不稳定且容易过拟合。与此同时，直接应用大语言模型（LLMs）往往忽视了表格数据的固有结构，导致性能欠佳。为了克服这些局限性，我们提出了FORESTLLM，这是一个新颖的框架，它将决策森林的结构归纳偏置与LLMs的语义推理能力统一起来。关键在于，FORESTLLM仅在训练阶段利用LLM，将其视为一个离线的模型设计器，将丰富的上下文知识编码到一个轻量级、可解释的森林模型中，从而在测试时无需进行LLM推理。我们的方法包含两个方面。首先，我们引入了一种语义分割准则，其中LLM基于标注和未标注数据的一致性来评估候选划分，从而能够在小样本监督下诱导出更鲁棒和泛化性更强的树结构。其次，我们提出了一种一次性上下文推理机制用于叶节点稳定化，其中LLM将决策路径及其支持示例提炼成一个简洁、确定性的预测，用语义信息丰富的输出替代了噪声较大的经验估计。在一系列多样化的小样本分类和回归基准测试中，FORESTLLM实现了最先进的性能。