Large language models (LLMs) encapsulate vast amounts of knowledge but still remain vulnerable to external misinformation. Existing research mainly studied this susceptibility behavior in a single-turn setting. However, belief can change during a multi-turn conversation, especially a persuasive one. Therefore, in this study, we delve into LLMs' susceptibility to persuasive conversations, particularly on factual questions that they can answer correctly. We first curate the Farm (i.e., Fact to Misinform) dataset, which contains factual questions paired with systematically generated persuasive misinformation. Then, we develop a testing framework to track LLMs' belief changes in a persuasive dialogue. Through extensive experiments, we find that LLMs' correct beliefs on factual knowledge can be easily manipulated by various persuasive strategies.

翻译：大语言模型（LLMs）封装了大量知识，但仍易受外部错误信息影响。现有研究主要关注单轮对话中的这种易感性行为。然而，信念可能在多轮对话——尤其是说服性对话——中发生改变。因此，本研究深入探讨LLMs对说服性对话的易感性，特别针对其本可正确回答的事实性问题。我们首先构建了Farm（即“事实到误导”）数据集，其中包含与系统生成的、具有说服力的错误信息配对的事实性问题。随后，我们开发了一个测试框架，用于追踪LLMs在说服性对话中的信念变化。通过大量实验，我们发现LLMs对事实性知识的正确信念极易受到各种说服策略的操纵。