With the increasing use of large-language models (LLMs) like ChatGPT, watermarking has emerged as a promising approach for tracing machine-generated content. However, research on LLM watermarking often relies on simple perplexity or diversity-based measures to assess the quality of watermarked text, which can mask important limitations in watermarking. Here we introduce two new easy-to-use methods for evaluating watermarking algorithms for LLMs: 1) evaluation by LLM-judger with specific guidelines; and 2) binary classification on text embeddings to distinguish between watermarked and unwatermarked text. We apply these methods to characterize the effectiveness of current watermarking techniques. Our experiments, conducted across various datasets, reveal that current watermarking methods are detectable by even simple classifiers, challenging the notion of watermarking subtlety. We also found, through the LLM judger, that watermarking impacts text quality, especially in degrading the coherence and depth of the response. Our findings underscore the trade-off between watermark robustness and text quality and highlight the importance of having more informative metrics to assess watermarking quality.

翻译：随着ChatGPT等大语言模型（LLM）的广泛应用，水印技术已成为追踪机器生成内容的一种有前景的方法。然而，当前关于LLM水印的研究通常依赖简单的困惑度或多样性指标来评估水印文本的质量，这可能会掩盖水印技术的重要局限性。本文提出两种易于使用的新方法来评估LLM水印算法：1）基于特定指导原则的LLM评判器评估；2）对文本嵌入进行二分类以区分水印文本与非水印文本。我们应用这些方法表征当前水印技术的有效性。跨多个数据集进行的实验表明，即使简单的分类器也能检测出现有水印方法，这挑战了水印隐蔽性的传统认知。通过LLM评判器我们还发现，水印会影响文本质量，尤其在降低回答的连贯性和深度方面。研究结果揭示了水印鲁棒性与文本质量之间的权衡，并强调了采用更具信息量的指标来评估水印质量的重要性。