Probing and enhancing large language models' reasoning capacity remains a crucial open question. Here we re-purpose the reverse dictionary task as a case study to probe LLMs' capacity for conceptual inference. We use in-context learning to guide the models to generate the term for an object concept implied in a linguistic description. Models robustly achieve high accuracy in this task, and their representation space encodes information about object categories and fine-grained features. Further experiments suggest that the conceptual inference ability as probed by the reverse-dictionary task predicts model's general reasoning performance across multiple benchmarks, despite similar syntactic generalization behaviors across models. Explorative analyses suggest that prompting LLMs with description$\Rightarrow$word examples may induce generalization beyond surface-level differences in task construals and facilitate models on broader commonsense reasoning problems.

翻译：探究并增强大语言模型的推理能力仍是关键未解难题。本文重新将反向词典任务设计为案例研究，用以探析大语言模型的概念推理能力。我们采用上下文学习引导模型根据语言描述中隐含的物体概念生成对应术语。模型在此任务中稳定实现高准确率，其表征空间编码了物体类别与细粒度特征信息。进一步实验表明，反向词典任务所探测的概念推理能力可预测模型在多个基准测试中的通用推理表现，尽管不同模型在句法泛化行为上表现相似。探索性分析提示，向大语言模型提供"描述⇒词汇"示例可能诱导其超越任务构式表层差异进行泛化，从而促进模型在更广泛的常识推理问题上的表现。