Watermarking approaches are proposed to identify if text being circulated is human or large language model (LLM) generated. The state-of-the-art watermarking strategy of Kirchenbauer et al. (2023a) biases the LLM to generate specific (``green'') tokens. However, determining the robustness of this watermarking method is an open problem. Existing attack methods fail to evade detection for longer text segments. We overcome this limitation, and propose {\em Self Color Testing-based Substitution (SCTS)}, the first ``color-aware'' attack. SCTS obtains color information by strategically prompting the watermarked LLM and comparing output tokens frequencies. It uses this information to determine token colors, and substitutes green tokens with non-green ones. In our experiments, SCTS successfully evades watermark detection using fewer number of edits than related work. Additionally, we show both theoretically and empirically that SCTS can remove the watermark for arbitrarily long watermarked text.

翻译：水印方法被提出来用于识别流通文本是由人类还是大型语言模型（LLM）生成。Kirchenbauer等人（2023a）提出的最先进水印策略会偏向LLM生成特定的（“绿色”）令牌。然而，确定该水印方法的鲁棒性仍是一个开放性问题。现有的攻击方法无法在较长文本片段中规避检测。我们克服了这一局限性，并提出“基于自颜色测试的替换方法”（Self Color Testing-based Substitution, SCTS）——这是首个“颜色感知”攻击。SCTS通过策略性地提示带有水印的LLM并比较输出令牌频率来获取颜色信息，利用该信息确定令牌颜色，并将绿色令牌替换为非绿色令牌。实验表明，SCTS在比相关工作更少的编辑次数下成功规避了水印检测。此外，我们从理论和实验两方面证明，SCTS能够移除任意长度水印文本中的水印。