Large language models (LLMs) can now generate and recognize text in a wide range of styles and genres, including highly specialized, creative genres like poetry. But what do LLMs really know about poetry? What can they know about poetry? We develop a task to evaluate how well LLMs recognize a specific aspect of poetry, poetic form, for more than 20 forms and formal elements in the English language. Poetic form captures many different poetic features, including rhyme scheme, meter, and word or line repetition. We use this task to reflect on LLMs' current poetic capabilities, as well as the challenges and pitfalls of creating NLP benchmarks for poetry and for other creative tasks. In particular, we use this task to audit and reflect on the poems included in popular pretraining datasets. Our findings have implications for NLP researchers interested in model evaluation, digital humanities and cultural analytics scholars, and cultural heritage professionals.

翻译：大型语言模型（LLMs）现已能够生成并识别多种风格与体裁的文本，包括诗歌这类高度专业化、创造性的体裁。但LLMs对诗歌究竟了解多少？它们能够掌握诗歌的哪些方面？我们设计了一项评估任务，用以检验LLMs对英语诗歌中二十余种诗歌形式与形式要素的识别能力。诗歌形式涵盖诸多诗学特征，包括韵律结构、格律以及词语或诗行的重复。借助该任务，我们反思了LLMs当前的诗学能力，以及为诗歌与其他创造性任务构建自然语言处理基准所面临的挑战与陷阱。特别地，我们运用该任务对主流预训练数据集中收录的诗歌进行了审计与反思。本研究对关注模型评估的自然语言处理研究者、数字人文与文化分析学者以及文化遗产专业人员具有重要启示。