Large Language Models (LLMs) encapsulate vast amounts of knowledge but still remain vulnerable to external misinformation. Existing research mainly studied this susceptibility behavior in a single-turn setting. However, belief can change during a multi-turn conversation, especially a persuasive one. Therefore, in this study, we delve into LLMs' susceptibility to persuasive conversations, particularly on factual questions that they can answer correctly. We first curate the Farm (i.e., Fact to Misinform) dataset, which contains factual questions paired with systematically generated persuasive misinformation. Then, we develop a testing framework to track LLMs' belief changes in a persuasive dialogue. Through extensive experiments, we find that LLMs' correct beliefs on factual knowledge can be easily manipulated by various persuasive strategies.

翻译：大语言模型（LLMs）封装了海量知识，但面对外部错误信息时仍显脆弱。现有研究主要聚焦于单轮交互场景下的这种易感性行为。然而，信念在多轮对话（尤其是说服性对话）中可能发生改变。因此，本研究深入探究LLMs对说服性对话的易感性——特别是针对它们能够正确回答的事实性问题。我们首先构建了Farm（即“事实到错误信息”）数据集，其中包含与系统性生成的虚假说服信息配对的事实性问题。随后，我们开发了一套测试框架，以追踪说服性对话中LLMs的信念变化。通过大量实验发现，LLMs对事实知识的正确信念极易被多种说服策略所操控。