Recent work shows that standard greedy-decoding extraction methods for quantifying memorization in LLMs miss how extraction risk varies across sequences. Probabilistic extraction -- computing the probability of generating a target suffix given a prefix under a decoding scheme -- addresses this, but is tractable only for verbatim memorization, missing near-verbatim instances that pose similar privacy and copyright risks. Quantifying near-verbatim extraction risk is expensive: the set of near-verbatim suffixes is combinatorially large, and reliable Monte Carlo (MC) estimation can require ~100,000 samples per sequence. To mitigate this cost, we introduce decoding-constrained beam search, which yields deterministic lower bounds on near-verbatim extraction risk at a cost comparable to ~20 MC samples per sequence. Across experiments, our approach surfaces information invisible to verbatim methods: many more extractable sequences, substantially larger per-sequence extraction mass, and patterns in how near-verbatim extraction risk manifests across model sizes and types of text.

翻译：近期研究表明，用于量化大语言模型记忆程度的贪心解码提取方法未能捕捉不同序列间提取风险的变化。概率化提取——即在给定解码方案下计算前缀生成目标后缀的概率——虽能解决这一问题，但仅适用于逐字记忆场景，无法应对同样具有隐私和版权风险的近逐字实例。量化近逐字提取风险成本高昂：近逐字后缀集合呈组合级规模，而可靠蒙特卡洛估计每个序列需约10万次采样。为降低该成本，我们提出解码约束束搜索方法，该方法能以每个序列约20次MC采样的成本获得近逐字提取风险的确定性下限。实验表明，本方法可揭示逐字方法无法感知的信息：更多可提取序列、显著增长的每序列提取质量，以及跨模型规模与文本类型的近逐字提取风险分布规律。