Zipf's law of abbreviation, the tendency of more frequent words to be shorter, is one of the most solid candidates for a linguistic universal, in the sense that it has the potential for being exceptionless or with a number of exceptions that is vanishingly small compared to the number of languages on Earth. Since Zipf's pioneering research, this law has been viewed as a manifestation of a universal principle of communication, i.e. the minimization of word lengths, to reduce the effort of communication. Here we revisit the concordance of written language with the law of abbreviation. Crucially, we provide wider evidence that the law holds also in speech (when word length is measured in time), in particular in 46 languages from 14 linguistic families. Agreement with the law of abbreviation provides indirect evidence of compression of languages via the theoretical argument that the law of abbreviation is a prediction of optimal coding. Motivated by the need of direct evidence of compression, we derive a simple formula for a random baseline indicating that word lengths are systematically below chance, across linguistic families and writing systems, and independently of the unit of measurement (length in characters or duration in time). Our work paves the way to measure and compare the degree of optimality of word lengths in languages.
翻译:齐普夫缩写定律,即高频词倾向于更短的现象,是语言普遍性最可靠的候选之一,因其具有无例外或例外数量相对于地球语言数量而言微乎其微的潜力。自齐普夫开创性研究以来,该定律被视为通信普遍原理(即最小化词长以降低通信成本)的体现。本文重新审视书面语言与缩写定律的一致性。关键在于,我们提供了更广泛的证据表明该定律在口语中(当词长以时间度量时)同样成立,尤其涵盖14个语系的46种语言。通过理论论证——缩写定律是最优编码的预测结果,与缩写定律的一致性为语言压缩提供了间接证据。受直接证据需求的驱动,我们推导出一个简单的随机基线公式,表明词长系统性低于随机水平,该现象跨越语系与文字系统,且独立于测量单位(字符长度或时间持续)。我们的工作为衡量与比较语言词长最优性程度铺平了道路。