Large language models (LLMs) are competitive with the state of the art on a wide range of sentence-level translation datasets. However, their ability to translate paragraphs and documents remains unexplored because evaluation in these settings is costly and difficult. We show through a rigorous human evaluation that asking the Gpt-3.5 (text-davinci-003) LLM to translate an entire literary paragraph (e.g., from a novel) at once results in higher-quality translations than standard sentence-by-sentence translation across 18 linguistically-diverse language pairs (e.g., translating into and out of Japanese, Polish, and English). Our evaluation, which took approximately 350 hours of effort for annotation and analysis, is conducted by hiring translators fluent in both the source and target language and asking them to provide both span-level error annotations as well as preference judgments of which system's translations are better. We observe that discourse-level LLM translators commit fewer mistranslations, grammar errors, and stylistic inconsistencies than sentence-level approaches. With that said, critical errors still abound, including occasional content omissions, and a human translator's intervention remains necessary to ensure that the author's voice remains intact. We publicly release our dataset and error annotations to spur future research on evaluation of document-level literary translation.

翻译：大型语言模型（LLM）在广泛句子级翻译数据集上已达到当前最优水平。然而，因其段落与文档翻译的评估成本高昂且实施困难，此类模型的段落与文档翻译能力仍鲜有探索。通过严格人工评估，我们发现：在18个语言多样性丰富的语对（例如日语、波兰语与英语之间的互译）中，要求GPT-3.5（text-davinci-003）模型一次性翻译完整文学段落（如小说选段）时，其译文质量显著优于逐句翻译方法。本评估耗时约350小时进行标注与分析，通过聘请精通源语言与目标语言的双语译者，要求其提供跨句级错误标注及译文质量偏好判断。结果显示，与句子级方法相比，语篇级LLM译者的误译、语法错误及文体不一致问题更少。尽管如此，关键错误仍普遍存在——包括偶发的内容缺失现象，且人类译者的干预对保留作者文风依旧不可或缺。我们已公开发布数据集与错误标注，以推动文档级文学翻译评估领域的未来研究。