Large Language Models (LLMs) have exhibited remarkable success in long-form context comprehension tasks. However, their capacity to generate long contents, such as reports and articles, remains insufficiently explored. Current benchmarks do not adequately assess LLMs' ability to produce informative and comprehensive content, necessitating a more rigorous evaluation approach. In this study, we introduce \textsc{ProxyQA}, a framework for evaluating long-form text generation, comprising in-depth human-curated \textit{meta-questions} spanning various domains. Each meta-question contains corresponding \textit{proxy-questions} with annotated answers. LLMs are prompted to generate extensive content in response to these meta-questions. Utilizing an evaluator and incorporating generated content as background context, \textsc{ProxyQA} evaluates the quality of generated content based on the evaluator's performance in answering the \textit{proxy-questions}. We examine multiple LLMs, emphasizing \textsc{ProxyQA}'s demanding nature as a high-quality assessment tool. Human evaluation demonstrates that evaluating through \textit{proxy-questions} is a highly self-consistent and human-criteria-correlated validation method. The dataset and leaderboard will be available at \url{https://github.com/Namco0816/ProxyQA}.

翻译：大语言模型（LLMs）在长文本理解任务中展现出显著成功，但其生成报告、文章等长内容的能力尚未得到充分探索。现有基准无法充分评估LLMs生成信息丰富且全面内容的能力，亟需更严格的评估方法。本研究提出\textsc{ProxyQA}——一个长文本生成评估框架，包含跨多领域的人工精心策划的\textit{元问题}，每个元问题对应具有标注答案的\textit{代理问题}。通过提示LLMs针对元问题生成扩展内容，\textsc{ProxyQA}利用评估器将生成内容作为背景上下文，基于评估器回答\textit{代理问题}的表现衡量生成内容质量。我们测试了多个LLMs，证明了\textsc{ProxyQA}作为高质量评估工具的高要求特性。人工评估表明，通过\textit{代理问题}进行评估是一种高度自洽且与人类标准相关的验证方法。数据集与排行榜将发布于\url{https://github.com/Namco0816/ProxyQA}。