Large Language Models (LLMs) have shown promising results in a variety of literary tasks, often using complex memorized details of narration and fictional characters. In this work, we evaluate the ability of Llama-3 at attributing utterances of direct-speech to their speaker in novels. The LLM shows impressive results on a corpus of 28 novels, surpassing published results with ChatGPT and encoder-based baselines by a large margin. We then validate these results by assessing the impact of book memorization and annotation contamination. We found that these types of memorization do not explain the large performance gain, making Llama-3 the new state-of-the-art for quotation attribution in English literature. We release publicly our code and data.

翻译：大型语言模型（LLM）在多种文学任务中展现出有前景的结果，通常利用其对叙事细节和虚构角色的复杂记忆。在本研究中，我们评估了Llama-3在小说中将直接引语归属至其说话者的能力。该LLM在包含28部小说的语料库上取得了令人印象深刻的结果，大幅超越了已发表的ChatGPT结果及基于编码器的基线模型。随后，我们通过评估书籍记忆和标注污染的影响来验证这些结果。研究发现，这些类型的记忆并不能解释其巨大的性能提升，从而使Llama-3成为英语文学引语归属任务的新技术标杆。我们公开了代码和数据。