Watermarking is an important tool for promoting the responsible use of language models (LMs). Existing watermarks insert a signal into generated tokens that either flags LM-generated text (zero-bit watermarking) or encodes more complex messages (multi-bit watermarking). Though a number of recent multi-bit watermarks insert several bits into text without perturbing average next-token predictions, they largely extend design principles from the zero-bit setting, such as encoding a single bit per token. Notably, the information-theoretic capacity of multi-bit watermarking -- the maximum number of bits per token that can be inserted and detected without changing average next-token predictions -- has remained unknown. We address this gap by deriving the first capacity characterization of multi-bit watermarks. Our results inform the design of ArcMark: a new watermark construction based on coding-theoretic principles that, under certain assumptions, achieves the capacity of the multi-bit watermark channel. In practice, ArcMark outperforms competing multi-bit watermarks in terms of bit rate per token and detection accuracy. Our work demonstrates that LM watermarking is fundamentally a channel coding problem, paving the way for principled coding-theoretic approaches to watermark design.

翻译：水印技术是促进语言模型（LM）负责任使用的重要工具。现有水印方法通过向生成标记中嵌入信号，实现两种功能：标记LM生成文本（零比特水印）或编码更复杂的消息（多比特水印）。尽管近期多项多比特水印技术能在不扰动平均下一标记预测的前提下向文本嵌入多个比特，但这些方法大多沿用了零比特水印的设计原则，例如每个标记仅编码单个比特。值得注意的是，多比特水印的信息论容量——即在保持平均下一标记预测不变的前提下，每个标记可嵌入和检测的最大比特数——始终未知。我们通过首次推导多比特水印的容量表征来填补这一空白。基于此理论成果，我们设计了ArcMark：一种基于编码理论原理的新型水印架构，在特定假设下可实现多比特水印信道的理论容量。实际应用中，ArcMark在每标记比特率和检测准确率方面均优于现有多比特水印方案。本研究表明LM水印本质上属于信道编码问题，为基于编码理论的系统性水印设计方法开辟了新路径。