Large Language Models (LLMs) encapsulate vast amounts of knowledge but still remain vulnerable to external misinformation. Existing research mainly studied this susceptibility behavior in a single-turn setting. However, belief can change during a multi-turn conversation, especially a persuasive one. Therefore, in this study, we delve into LLMs' susceptibility to persuasive conversations, particularly on factual questions that they can answer correctly. We first curate the Farm (i.e., Fact to Misinform) dataset, which contains factual questions paired with systematically generated persuasive misinformation. Then, we develop a testing framework to track LLMs' belief changes in a persuasive dialogue. Through extensive experiments, we find that LLMs' correct beliefs on factual knowledge can be easily manipulated by various persuasive strategies.

翻译：大语言模型（LLMs）封装了海量知识，但仍易受外部错误信息影响。现有研究主要关注其在单轮交互中的脆弱性行为。然而，在多轮对话（尤其是说服性对话）中，信念可能发生改变。因此，本研究深入探究LLMs对说服性对话的敏感性，特别是针对其本可正确回答的事实性问题。我们首先构建了Farm（即事实到误导）数据集，其中包含与系统性生成的误导性说服信息配对的事实性问题。随后开发了测试框架，用于追踪LLMs在说服性对话中的信念变化。通过大量实验发现，LLMs对事实性知识的正确信念易被多种说服策略所操纵。