Phonology, the study of speech's structure and pronunciation rules, is a critical yet often overlooked component in Large Language Model (LLM) research. LLMs are widely used in various downstream applications that leverage phonology such as educational tools and poetry generation. Moreover, LLMs can potentially learn imperfect associations between orthographic and phonological forms from the training data. Thus, it is imperative to benchmark the phonological skills of LLMs. To this end, we present PhonologyBench, a novel benchmark consisting of three diagnostic tasks designed to explicitly test the phonological skills of LLMs in English: grapheme-to-phoneme conversion, syllable counting, and rhyme word generation. Despite having no access to speech data, LLMs showcased notable performance on the PhonologyBench tasks. However, we observe a significant gap of 17% and 45% on Rhyme Word Generation and Syllable counting, respectively, when compared to humans. Our findings underscore the importance of studying LLM performance on phonological tasks that inadvertently impact real-world applications. Furthermore, we encourage researchers to choose LLMs that perform well on the phonological task that is closely related to the downstream application since we find that no single model consistently outperforms the others on all the tasks.

翻译：音系学作为研究语音结构与发音规则的学科，在大语言模型研究中虽至关重要却常被忽视。当前大语言模型已广泛应用于教育工具、诗歌生成等依赖音系知识的下游任务，且模型可能从训练数据中习得拼写与语音形式间的非完美关联，因此系统性地评估大语言模型的音系能力显得尤为必要。为此，我们构建了PhonologyBench基准测试集，包含三项专门针对英语音系能力的诊断任务：字素-音素转换、音节计数与押韵词生成。实验表明，尽管缺乏语音数据输入，大语言模型在PhonologyBench任务中仍展现出显著性能。但与人类表现相比，模型在押韵词生成和音节计数任务中分别存在17%和45%的显著差距。本研究发现凸显了系统研究大语言模型音系任务表现的重要性——这种能力会潜移默化地影响实际应用效果。此外，由于未发现任何单一模型在所有任务上保持绝对优势，我们建议研究者根据下游应用场景中紧密相关的音系任务特征，选择在该任务上表现优异的大语言模型。