Large Language Models (LLMs) have exhibited remarkable success in long-form context comprehension tasks. However, their capacity to generate long contents, such as reports and articles, remains insufficiently explored. Current benchmarks do not adequately assess LLMs' ability to produce informative and comprehensive content, necessitating a more rigorous evaluation approach. In this study, we introduce \textsc{ProxyQA}, a framework for evaluating long-form text generation, comprising in-depth human-curated \textit{meta-questions} spanning various domains. Each meta-question contains corresponding \textit{proxy-questions} with annotated answers. LLMs are prompted to generate extensive content in response to these meta-questions. Utilizing an evaluator and incorporating generated content as background context, \textsc{ProxyQA} evaluates the quality of generated content based on the evaluator's performance in answering the \textit{proxy-questions}. We examine multiple LLMs, emphasizing \textsc{ProxyQA}'s demanding nature as a high-quality assessment tool. Human evaluation demonstrates that evaluating through \textit{proxy-questions} is a highly self-consistent and human-criteria-correlated validation method. The dataset and leaderboard will be available at \url{https://github.com/Namco0816/ProxyQA}.

翻译：大语言模型（LLMs）在长文本理解任务中展现出卓越的性能，但其生成报告、文章等长篇幅内容的能力尚未得到充分探索。现有基准测试未能充分评估LLMs生成信息丰富且全面内容的能力，亟需更严格的评估方法。本研究提出\textsc{ProxyQA}框架，用于评估长篇文本生成能力，该框架包含人工精心设计的跨领域\textit{元问题}。每个元问题对应若干带标注答案的\textit{代理问题}。模型需根据元问题生成扩展性内容，通过评估器将生成内容作为背景上下文，基于评估器回答\textit{代理问题}的表现来评判内容质量。我们通过多个LLMs验证了\textsc{ProxyQA}作为高质量评估工具的严格性。人工评估表明，基于\textit{代理问题}的评估方法具有高度自一致性，且与人类评判标准高度相关。数据集与排行榜将于\url{https://github.com/Namco0816/ProxyQA}公开。