Counterfactual text generation aims to minimally change a text, such that it is classified differently. Judging advancements in method development for counterfactual text generation is hindered by a non-uniform usage of data sets and metrics in related work. We propose CEval, a benchmark for comparing counterfactual text generation methods. CEval unifies counterfactual and text quality metrics, includes common counterfactual datasets with human annotations, standard baselines (MICE, GDBA, CREST) and the open-source language model LLAMA-2. Our experiments found no perfect method for generating counterfactual text. Methods that excel at counterfactual metrics often produce lower-quality text while LLMs with simple prompts generate high-quality text but struggle with counterfactual criteria. By making CEval available as an open-source Python library, we encourage the community to contribute more methods and maintain consistent evaluation in future work.

翻译：反事实文本生成旨在对文本进行最小化修改，使其分类结果发生改变。由于相关工作中数据集与评估指标的使用不统一，该方法开发进展的判断受到阻碍。我们提出CEval基准，用于比较反事实文本生成方法。CEval统一了反事实指标与文本质量指标，包含带有人工标注的常用反事实数据集、标准基线方法（MICE、GDBA、CREST）及开源语言模型LLAMA-2。实验发现，尚无完美方法能生成反事实文本：在反事实指标上表现优异的方法常生成低质量文本，而采用简单提示的LLM虽能生成高质量文本，却难以满足反事实标准。通过将CEval作为开源Python库发布，我们鼓励学界贡献更多方法，并在未来研究中保持评估一致性。