Large Language Models (LLMs) encapsulate vast amounts of knowledge but still remain vulnerable to external misinformation. Existing research mainly studied this susceptibility behavior in a single-turn setting. However, belief can change during a multi-turn conversation, especially a persuasive one. Therefore, in this study, we delve into LLMs' susceptibility to persuasive conversations, particularly on factual questions that they can answer correctly. We first curate the Farm (i.e., Fact to Misinform) dataset, which contains factual questions paired with systematically generated persuasive misinformation. Then, we develop a testing framework to track LLMs' belief changes in a persuasive dialogue. Through extensive experiments, we find that LLMs' correct beliefs on factual knowledge can be easily manipulated by various persuasive strategies.

翻译：大语言模型（LLMs）封装了海量知识，但仍易受外部错误信息影响。现有研究主要关注单轮交互中的这种易感性行为。然而，在多轮对话——尤其是说服性对话——中，信念可能发生改变。因此，本研究深入探讨了LLMs对说服性对话的易感性，特别是针对其能正确回答的事实性问题。我们首先构建了Farm（即事实向错误信息转化）数据集，其中包含事实性问答及其系统生成的具有说服力的错误信息。随后，我们开发了一个测试框架，用于追踪说服性对话中LLMs的信念变化。通过大量实验，我们发现，LLMs对事实性知识的正确信念容易被多种说服策略操纵。