While large language models (LLMs) excel in various natural language tasks in English, their performance in lower-resourced languages like Hebrew, especially for generative tasks such as abstractive summarization, remains unclear. The high morphological richness in Hebrew adds further challenges due to the ambiguity in sentence comprehension and the complexities in meaning construction. In this paper, we address this resource and evaluation gap by introducing HeSum, a novel benchmark specifically designed for abstractive text summarization in Modern Hebrew. HeSum consists of 10,000 article-summary pairs sourced from Hebrew news websites written by professionals. Linguistic analysis confirms HeSum's high abstractness and unique morphological challenges. We show that HeSum presents distinct difficulties for contemporary state-of-the-art LLMs, establishing it as a valuable testbed for generative language technology in Hebrew, and MRLs generative challenges in general.

翻译：尽管大型语言模型（LLM）在英语的多种自然语言任务中表现出色，但它们在希伯来语等资源较少的语言中的性能，特别是对于抽象摘要等生成任务，仍不明确。希伯来语高度的形态丰富性，由于句子理解的歧义性和意义构建的复杂性，带来了进一步的挑战。本文通过引入HeSum来弥补这一资源和评估缺口，这是一个专门为现代希伯来语抽象文本摘要设计的新型基准。HeSum包含10,000个文章-摘要对，源自专业人士撰写的希伯来语新闻网站。语言分析证实了HeSum具有高度的抽象性和独特的形态学挑战。我们表明，HeSum对当代最先进的LLM构成了显著困难，从而确立了其作为希伯来语生成语言技术以及形态丰富语言（MRL）生成挑战通用测试平台的价值。