The impact of Large Language Models (LLMs) has extended into literary domains. However, existing evaluation metrics for literature prioritize mechanical accuracy over artistic expression and tend to overrate machine translation as being superior to human translation from experienced professionals. In the long run, this bias could result in an irreversible decline in translation quality and cultural authenticity. In response to the urgent need for a specialized literary evaluation metric, we introduce LITRANSPROQA, a novel, reference-free, LLM-based question-answering framework designed for literary translation evaluation. LITRANSPROQA integrates humans in the loop to incorporate insights from professional literary translators and researchers, focusing on critical elements in literary quality assessment such as literary devices, cultural understanding, and authorial voice. Our extensive evaluation shows that while literary-finetuned XCOMET-XL yields marginal gains, LITRANSPROQA substantially outperforms current metrics, achieving up to 0.07 gain in correlation and surpassing the best state-of-the-art metrics by over 15 points in adequacy assessments. Incorporating professional translator insights as weights further improves performance, highlighting the value of translator inputs. Notably, LITRANSPROQA reaches an adequacy performance comparable to trained linguistic student evaluators, though it still falls behind experienced professional translators. LITRANSPROQA shows broad applicability to open-source models like LLaMA3.3-70b and Qwen2.5-32b, indicating its potential as an accessible and training-free tool for evaluating literary translations that require local processing due to copyright or ethical considerations.

翻译：大语言模型（LLMs）的影响已延伸至文学领域。然而，现有的文学翻译评估指标往往侧重于机械准确性而非艺术表达，并倾向于高估机器翻译，认为其优于经验丰富的专业译者的人工翻译。长远来看，这种偏见可能导致翻译质量与文化真实性的不可逆衰退。为应对对专业化文学评估指标的迫切需求，我们提出了LiTransProQA，一种新颖的、无参考的、基于大语言模型的问答框架，专为文学翻译评估而设计。LiTransProQA采用人机协同机制，整合了专业文学译者与研究者的洞见，重点关注文学质量评估中的关键要素，如文学手法、文化理解和作者风格。我们的大规模评估表明，虽然经过文学微调的XCOMET-XL仅带来边际提升，但LiTransProQA显著优于现有指标，在相关性上最高获得0.07的提升，并在充分性评估中超越最佳前沿指标超过15分。将专业译者洞见作为权重进一步提升了性能，凸显了译者输入的价值。值得注意的是，LiTransProQA在充分性方面的表现已达到与经过训练的语言学学生评估者相当的水平，尽管仍落后于经验丰富的专业译者。LiTransProQA对LLaMA3.3-70b和Qwen2.5-32b等开源模型展现出广泛的适用性，表明其有潜力成为一种易于获取且无需训练的工具，用于评估因版权或伦理考量需进行本地处理的文学翻译。