We investigate the degree to which human plausibility judgments of multiple-choice commonsense benchmark answers are subject to influence by (im)plausibility arguments for or against an answer, in particular, using rationales generated by LLMs. We collect 3,000 plausibility judgments from humans and another 13,600 judgments from LLMs. Overall, we observe increases and decreases in mean human plausibility ratings in the presence of LLM-generated PRO and CON rationales, respectively, suggesting that, on the whole, human judges find these rationales convincing. Experiments with LLMs reveal similar patterns of influence. Our findings demonstrate a novel use of LLMs for studying aspects of human cognition, while also raising practical concerns that, even in domains where humans are ``experts'' (i.e., common sense), LLMs have the potential to exert considerable influence on people's beliefs.

翻译：本研究探讨了人类对多项选择常识基准答案的合理性判断在多大程度上受到支持或反对答案的（非）合理性论证的影响，特别是使用大型语言模型生成的推理依据。我们收集了3,000份人类合理性判断和另外13,600份来自大型语言模型的判断。总体而言，在存在大型语言模型生成的支持性与反对性推理时，人类合理性评分的平均值分别呈现上升和下降趋势，这表明人类评判者整体上认为这些推理具有说服力。使用大型语言模型进行的实验也显示出类似的影响模式。我们的研究结果展示了利用大型语言模型研究人类认知特征的新途径，同时也提出了实际关切：即使在人类作为"专家"的领域（即常识领域），大型语言模型仍可能对人们的信念产生显著影响。