Despite -- or perhaps because of -- their simplicity, n-grams, or contiguous sequences of tokens, have been used with great success in computational linguistics since their introduction in the late 20th century. Recast as k-mers, or contiguous sequences of monomers, they have also found applications in computational biology. When applied to the analysis of texts, n-grams usually take the form of sequences of words. But if we try to apply this model to the analysis of Sanskrit texts, we are faced with the arduous task of, firstly, resolving sandhi to split a phrase into words, and, secondly, splitting long compounds into their components. This paper presents a simpler method of tokenizing a Sanskrit text for n-grams, by using n-aksaras, or contiguous sequences of aksaras. This model reduces the need for sandhi resolution, making it much easier to use on raw text. It is also possible to use this model on Sanskrit-adjacent texts, e.g., a Tamil commentary on a Sanskrit text. As a test case, the commentaries on Amarakosa 1.0.1 have been modelled as n-aksaras, showing patterns of text reuse across ten centuries and nine languages. Some initial observations are made concerning Buddhist commentarial practices.
翻译:尽管——或许正因其简洁性——n-gram(即连续标记序列)自20世纪末引入计算语言学以来,已被成功应用于该领域。若将其重构为k-mer(即连续单体序列),则在计算生物学中也得到了应用。在文本分析中,n-gram通常采用词序列的形式。但若试图将该模型应用于梵文文本分析,首先需要解决连音变调(sandhi)问题以将短语切分为词汇,其次要分解冗长的复合词。本文提出一种更简单的梵文文本n-gram分词方法:使用n-akṣara——即连续音节序列。该模型降低了对连音变调解析的依赖,使其更易应用于原始文本。此外,该模型还可用于与梵文相关的文本(例如,针对梵文文本的泰米尔语注释)。以《Amarakośa》1.0.1的注释文本为测试案例,通过n-akṣara建模揭示了跨越十个世纪、九种语言的文本复用模式,并针对佛教注释实践提出初步观察结论。