The diversity across outputs generated by LLMs shapes perception of their quality and utility. High lexical diversity is often desirable, but there is no standard method to measure this property. Templated answer structures and ``canned'' responses across different documents are readily noticeable, but difficult to visualize across large corpora. This work aims to standardize measurement of text diversity. Specifically, we empirically investigate the convergent validity of existing scores across English texts, and we release diversity, an open-source Python package for measuring and extracting repetition in text. We also build a platform based on diversity for users to interactively explore repetition in text. We find that fast compression algorithms capture information similar to what is measured by slow-to-compute $n$-gram overlap homogeneity scores. Further, a combination of measures -- compression ratios, self-repetition of long $n$-grams, and Self-BLEU and BERTScore -- are sufficient to report, as they have low mutual correlation with each other.

翻译：大型语言模型生成输出的多样性直接影响其质量和实用性的感知。高词汇多样性通常是可取的，但目前尚无测量此特性的标准方法。模板化答案结构和跨文档的“固定”响应模式易于察觉，但难以在大规模语料库中进行可视化分析。本研究旨在标准化文本多样性的测量。具体而言，我们通过实证研究检验了现有评分方法在英文文本上的收敛效度，并发布了diversity——一个用于测量和提取文本重复性的开源Python工具包。同时，我们基于diversity构建了交互式平台，使用户能够直观探索文本中的重复模式。研究发现，快速压缩算法捕获的信息与计算耗时的$n$元语法重叠同质性评分所测量的信息具有相似性。此外，综合采用压缩比、长$n$元语法自重复率、Self-BLEU与BERTScore的组合指标足以有效表征多样性，因为这些指标彼此间具有较低的相关性。