Large language models (LLMs) are increasingly used for everyday communication tasks, including drafting interpersonal messages intended to influence and persuade. Prior work has shown that LLMs can successfully persuade humans and amplify persuasive language. It is therefore essential to understand how user instructions affect the generation of persuasive language, and to understand whether the generated persuasive language differs, for example, when targeting different groups. In this work, we propose a framework for evaluating how persuasive language generation is affected by recipient gender, sender intent, or output language. We evaluate 13 LLMs and 16 languages using pairwise prompt instructions. We evaluate model responses on 19 categories of persuasive language using an LLM-as-judge setup grounded in social psychology and communication science. Our results reveal significant gender differences in the persuasive language generated across all models. These patterns reflect biases consistent with gender-stereotypical linguistic tendencies documented in social psychology and sociolinguistics.

翻译：大型语言模型（LLM）正日益被用于日常沟通任务，包括起草旨在影响和说服的人际信息。先前研究表明，LLM能够成功说服人类并强化说服性语言。因此，理解用户指令如何影响说服性语言的生成至关重要，同时需要探究生成的说服性语言是否存在差异——例如针对不同群体时。本研究提出一个评估框架，用于分析接收者性别、发送者意图或输出语言如何影响说服性语言的生成。我们通过配对提示指令评估了13个LLM和16种语言，并基于社会心理学与传播学理论，采用LLM作为评判者的设置，从19个说服性语言类别对模型响应进行评估。研究结果显示，所有模型生成的说服性语言均存在显著的性别差异。这些模式反映出的偏见与社会心理学及社会语言学中记载的性别刻板语言倾向具有一致性。