Syllogistic reasoning is crucial for Natural Language Inference (NLI). This capability is particularly significant in specialized domains such as biomedicine, where it can support automatic evidence interpretation and scientific discovery. This paper presents SylloBio-NLI, a novel framework that leverages external ontologies to systematically instantiate diverse syllogistic arguments for biomedical NLI. We employ SylloBio-NLI to evaluate Large Language Models (LLMs) on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. Extensive experiments reveal that biomedical syllogistic reasoning is particularly challenging for zero-shot LLMs, which achieve an average accuracy between 70% on generalized modus ponens and 23% on disjunctive syllogism. At the same time, we found that few-shot prompting can boost the performance of different LLMs, including Gemma (+14%) and LLama-3 (+43%). However, a deeper analysis shows that both techniques exhibit high sensitivity to superficial lexical variations, highlighting a dependency between reliability, models' architecture, and pre-training regime. Overall, our results indicate that, while in-context examples have the potential to elicit syllogistic reasoning in LLMs, existing models are still far from achieving the robustness and consistency required for safe biomedical NLI applications.

翻译：三段论推理对于自然语言推理至关重要。这种能力在生物医学等专业领域中尤为重要，它可以支持自动证据解读和科学发现。本文提出了SylloBio-NLI，这是一个新颖的框架，利用外部本体论来系统化地实例化用于生物医学自然语言推理的多样化三段论论证。我们运用SylloBio-NLI来评估大语言模型在识别有效结论和提取支持证据方面的能力，这些证据基于28种利用人类基因组通路实例化的三段论图式。大量实验表明，生物医学三段论推理对于零样本大语言模型尤其具有挑战性，其在广义肯定前件式上的平均准确率约为70%，而在析取三段论上仅为23%。同时，我们发现少量样本提示可以提升不同大语言模型的性能，包括Gemma（+14%）和LLama-3（+43%）。然而，更深入的分析显示，这两种技术对表面的词汇变化都表现出高度敏感性，凸显了可靠性、模型架构与预训练方案之间的依赖性。总体而言，我们的结果表明，尽管上下文示例有潜力激发大语言模型的三段论推理能力，但现有模型仍远未达到安全的生物医学自然语言推理应用所需的鲁棒性和一致性。