This paper introduces ChatbotManip, a novel dataset for studying manipulation in Chatbots. It contains simulated generated conversations between a chatbot and a (simulated) user, where the chatbot is explicitly asked to showcase manipulation tactics, persuade the user towards some goal, or simply be helpful. We consider a diverse set of chatbot manipulation contexts, from consumer and personal advice to citizen advice and controversial proposition argumentation. Each conversation is annotated by human annotators for both general manipulation and specific manipulation tactics. Our research reveals three key findings. First, Large Language Models (LLMs) can be manipulative when explicitly instructed, with annotators identifying manipulation in approximately 84\% of such conversations. Second, even when only instructed to be ``persuasive'' without explicit manipulation prompts, LLMs frequently default to controversial manipulative strategies, particularly gaslighting and fear enhancement. Third, small fine-tuned open source models, such as BERT+BiLSTM have a performance comparable to zero-shot classification with larger models like Gemini 2.5 pro in detecting manipulation, but are not yet reliable for real-world oversight. Our work provides important insights for AI safety research and highlights the need of addressing manipulation risks as LLMs are increasingly deployed in consumer-facing applications.

翻译：本文提出ChatbotManip——一个用于研究聊天机器人操纵行为的新颖数据集。该数据集包含聊天机器人与（模拟）用户之间生成的模拟对话，其中要求聊天机器人明确展示操纵策略、说服用户实现特定目标，或仅提供帮助。我们考虑了多样化的聊天机器人操纵场景，涵盖消费者建议、个人建议、公民建议及争议性命题论证。每条对话均由人类标注员对通用操纵行为和具体操纵策略进行标注。研究揭示了三个关键发现：第一，当被明确指示时，大语言模型（LLMs）可能表现出操纵性，标注员在约84%的此类对话中识别出操纵行为；第二，即使仅被指示进行“说服”而未提供明确操纵提示，LLMs也频繁默认采用有争议的操纵策略，尤其是煤气灯效应和恐惧强化；第三，小型微调开源模型（如BERT+BiLSTM）在检测操纵行为方面，其性能可与大型模型（如Gemini 2.5 Pro）的零样本分类相媲美，但仍不足以用于实际监管。我们的工作为人工智能安全研究提供了重要见解，并强调随着LLMs在面向消费者应用中日益普及，亟需解决操纵风险问题。