This work explores the capability of conversational chatbots powered by large language models (LLMs), to understand and characterize predicate symmetry, a cognitive linguistic function traditionally believed to be an inherent human trait. Leveraging in-context learning (ICL), a paradigm shift enabling chatbots to learn new tasks from prompts without re-training, we assess the symmetrical reasoning of five chatbots: ChatGPT 4, Huggingface chat AI, Microsoft's Copilot AI, LLaMA through Perplexity, and Gemini Advanced. Using the Symmetry Inference Sentence (SIS) dataset by Tanchip et al. (2020), we compare chatbot responses against human evaluations to gauge their understanding of predicate symmetry. Experiment results reveal varied performance among chatbots, with some approaching human-like reasoning capabilities. Gemini, for example, reaches a correlation of 0.85 with human scores, while providing a sounding justification for each symmetry evaluation. This study underscores the potential and limitations of LLMs in mirroring complex cognitive processes as symmetrical reasoning.

翻译：本研究探讨了基于大型语言模型（LLM）的对话机器人理解与表征谓词对称性——这一传统上被认为是人类固有特质的认知语言功能——的能力。利用上下文学习（ICL）这一使机器人能够从提示中学习新任务而无需重新训练的模式转变，我们评估了五款机器人的对称推理能力：ChatGPT 4、Huggingface chat AI、微软的Copilot AI、通过Perplexity访问的LLaMA以及Gemini Advanced。使用Tanchip等人（2020）提出的对称推理语句（SIS）数据集，我们将机器人的回答与人类评估进行比较，以衡量其对谓词对称性的理解。实验结果显示，各机器人表现不一，其中一些已接近类人的推理能力。例如，Gemini与人类评分的相关性达到0.85，并为每次对称性评估提供了合理的论证。本研究揭示了LLM在模拟对称推理这类复杂认知过程中的潜力与局限性。