We present a novel dataset for the controlled composition of counterarguments designed for further applications in argument refining, mining, and evaluation. Our dataset constitutes enriched counter-arguments to posts in the Reddit ChangeMyView dataset that are integrated with evidence retrieved from high-quality sources and generated based on user preferences, adjusting the critical attributes of evidence and argument style. The resultant Counterfire corpus comprises arguments generated from GPT-3.5 turbo, Koala, and PaLM 2 models and two of their finetuned variants (N = 32,000). Model evaluation indicates strong paraphrasing abilities with evidence, albeit limited word overlap, while demonstrating high style integration (0.9682 for 'reciprocity'), showing the ability of LLM to assimilate diverse styles. Of all models, GPT-3.5 turbo showed the highest scores in argument quality evaluation, showing consistent accuracy (score >0.8). In further analyses, reciprocity-style counterarguments display higher counts in most categories, possibly indicating a more creatively persuasive use of evidence. In contrast, human-written counterarguments exhibited greater argumentative richness and diversity across categories. Despite human-written arguments being favored as the most persuasive in human evaluation, the 'No Style' generated text surprisingly exhibited the highest score, prompting further exploration and investigation on the trade-offs in generation for facts and style.

翻译：我们提出了一种用于控制性组合反驳的新数据集，旨在进一步应用于论点精炼、挖掘和评估。该数据集包含对Reddit ChangeMyView数据集中帖子的增强型反驳，这些反驳整合了从高质量来源检索的证据，并根据用户偏好生成，调整了证据和论点风格的关键属性。由此产生的Counterfire语料库包含来自GPT-3.5 turbo、Koala和PaLM 2模型及其两种微调变体生成的论点（N=32,000）。模型评估表明，虽然词汇重叠有限，但模型在结合证据方面表现出强大的释义能力，同时展现出高度风格整合能力（“互惠性”风格为0.9682），显示了大型语言模型吸收多种风格的能力。在所有模型中，GPT-3.5 turbo在论点质量评估中得分最高，表现出持续准确性（得分>0.8）。进一步分析显示，互惠性风格的反驳在大多数类别中数量更高，可能表明对证据更具创造性的说服性使用。相比之下，人工撰写的反驳在各类别中表现出更强的论证丰富性和多样性。尽管人工撰写的论点在人类评估中被认为最具说服力，但“无风格”生成的文本出人意料地获得了最高分，这促使我们对事实与风格生成中的权衡进行进一步探索和研究。