Data containing personal information is increasingly used to train, fine-tune, or query Large Language Models (LLMs). Text is typically scrubbed of identifying information prior to use, often with tools such as Microsoft's Presidio or Anthropic's PII purifier. These tools have traditionally been evaluated on their ability to remove specific identifiers (e.g., names), yet their effectiveness at preventing re-identification remains unclear. We introduce RAT-Bench, a comprehensive benchmark for text anonymization tools based on re-identification risk. Using U.S. demographic statistics, we generate synthetic text containing various direct and indirect identifiers across domains, languages, and difficulty levels. We evaluate a range of NER- and LLM-based text anonymization tools and, based on the attributes an LLM-based attacker is able to correctly infer from the anonymized text, we report the risk of re-identification in the U.S. population, while properly accounting for the disparate impact of identifiers. We find that, while capabilities vary widely, even the best tools are far from perfect in particular when direct identifiers are not written in standard ways and when indirect identifiers enable re-identification. Overall we find LLM-based anonymizers, including new iterative anonymizers, to provide a better privacy-utility trade-off albeit at a higher computational cost. Importantly, we also find them to work well across languages. We conclude with recommendations for future anonymization tools and will release the benchmark and encourage community efforts to expand it, in particular to other geographies.

翻译：包含个人信息的数据越来越多地被用于训练、微调或查询大型语言模型（LLM）。在使用前，文本通常需经过识别信息擦除处理，常用工具包括微软的Presidio或Anthropic的PII净化器。传统上对这些工具的评估侧重于其移除特定标识符（如姓名）的能力，但其在防止再识别方面的有效性仍不明确。我们提出了RAT-Bench，一个基于再识别风险的文本匿名化工具综合基准。利用美国人口统计数据，我们生成了包含跨领域、跨语言及不同难度级别的各类直接与间接标识符的合成文本。我们评估了一系列基于命名实体识别（NER）和LLM的文本匿名化工具，并基于LLM攻击者能够从匿名化文本中正确推断出的属性，报告了在美国人口中的再识别风险，同时恰当考虑了标识符的差异性影响。我们发现，尽管各工具能力差异显著，但即使最佳工具也远非完美，尤其是在直接标识符以非标准方式书写以及间接标识符可能导致再识别的情况下。总体而言，我们发现基于LLM的匿名化工具（包括新型迭代式匿名化器）能提供更好的隐私-效用权衡，尽管计算成本更高。重要的是，我们还发现它们在不同语言中均表现良好。最后，我们为未来匿名化工具提出了建议，并将发布此基准，鼓励社区努力扩展其范围，特别是扩展到其他地理区域。