Large Language Models (LLMs) encapsulate vast amounts of knowledge but still remain vulnerable to external misinformation. Existing research mainly studied this susceptibility behavior in a single-turn setting. However, belief can change during a multi-turn conversation, especially a persuasive one. Therefore, in this study, we delve into LLMs' susceptibility to persuasive conversations, particularly on factual questions that they can answer correctly. We first curate the Farm (i.e., Fact to Misinform) dataset, which contains factual questions paired with systematically generated persuasive misinformation. Then, we develop a testing framework to track LLMs' belief changes in a persuasive dialogue. Through extensive experiments, we find that LLMs' correct beliefs on factual knowledge can be easily manipulated by various persuasive strategies.

翻译：大语言模型（LLMs）虽封装了海量知识，却仍易受外部错误信息影响。现有研究主要在单轮对话场景中探究这一易感性行为。然而，在多轮对话——尤其是说服性对话——中，信念可能发生改变。因此，本研究深入探讨LLMs对说服性对话的易感性，特别是针对其能正确回答的事实性问题。我们首先构建了Farm（即“事实到错误信息”）数据集，其中包含与系统生成的说服性错误信息配对的事实性问题。随后，我们开发了一个测试框架，用以追踪LLMs在说服性对话中的信念变化。通过大量实验，我们发现：LLMs对事实性知识的正确信念易被多种说服策略所操纵。