The capabilities of large language models (LLMs) have raised concerns about their potential to create and propagate convincing narratives. Here, we study their performance in detecting convincing arguments to gain insights into LLMs' persuasive capabilities without directly engaging in experimentation with humans. We extend a dataset by Durmus and Cardie (2018) with debates, votes, and user traits and propose tasks measuring LLMs' ability to (1) distinguish between strong and weak arguments, (2) predict stances based on beliefs and demographic characteristics, and (3) determine the appeal of an argument to an individual based on their traits. We show that LLMs perform on par with humans in these tasks and that combining predictions from different LLMs yields significant performance gains, surpassing human performance. The data and code released with this paper contribute to the crucial effort of continuously evaluating and monitoring LLMs' capabilities and potential impact. (https://go.epfl.ch/persuasion-llm)

翻译：大型语言模型（LLM）的能力引发了人们对其可能创造和传播具有说服力叙事的担忧。本文通过研究其在检测具有说服力论点方面的表现，旨在深入理解LLM的说服能力，而无需直接进行涉及人类的实验。我们扩展了Durmus和Cardie（2018）的数据集，加入了辩论、投票和用户特征，并提出了衡量LLM能力的任务：（1）区分强论点与弱论点；（2）基于信念和人口统计学特征预测立场；（3）根据个体特征判断论点对其的吸引力。研究表明，LLM在这些任务上的表现与人类相当，并且结合不同LLM的预测能带来显著的性能提升，甚至超越人类表现。本文发布的数据和代码有助于持续评估和监测LLM的能力及其潜在影响这一重要工作。（https://go.epfl.ch/persuasion-llm）