Recent advancements in transformer-based models have initiated research interests in investigating their ability to learn to perform reasoning tasks. However, most of the contexts used for this purpose are in practice very simple: generated from short (fragments of) first-order logic sentences with only a few logical operators and quantifiers. In this work, we construct the natural language dataset, DELTA$_D$, using the description logic language $\mathcal{ALCQ}$. DELTA$_D$ contains 384K examples, and increases in two dimensions: i) reasoning depth, and ii) linguistic complexity. In this way, we systematically investigate the reasoning ability of a supervised fine-tuned DeBERTa-based model and of two large language models (GPT-3.5, GPT-4) with few-shot prompting. Our results demonstrate that the DeBERTa-based model can master the reasoning task and that the performance of GPTs can improve significantly even when a small number of samples is provided (9 shots). We open-source our code and datasets.

翻译：近年来，基于Transformer的模型所取得的进展，引发了学界对其学习执行推理任务能力的探究兴趣。然而，目前用于此类研究的大多数上下文实际上都非常简单：它们源自仅包含少量逻辑运算符和量词的一阶逻辑短句（片段）。本研究利用描述逻辑语言$\mathcal{ALCQ}$构建了自然语言数据集DELTA$_D$。该数据集包含38.4万个样本，并在两个维度上呈现递增特征：i) 推理深度，ii) 语言复杂度。通过这种设计，我们系统探究了经过监督微调的DeBERTa模型以及采用少样本提示的两个大型语言模型（GPT-3.5、GPT-4）的推理能力。实验结果表明，基于DeBERTa的模型能够掌握推理任务，而GPT系列模型即使仅提供少量样本（9次提示），其性能也能显著提升。我们已将代码和数据集开源。