Modern generative search engines enhance the reliability of large language model (LLM) responses by providing cited evidence. However, evaluating the answer's attribution, i.e., whether every claim within the generated responses is fully supported by its cited evidence, remains an open problem. This verification, traditionally dependent on costly human evaluation, underscores the urgent need for automatic attribution evaluation methods. To bridge the gap in the absence of standardized benchmarks for these methods, we present AttributionBench, a comprehensive benchmark compiled from various existing attribution datasets. Our extensive experiments on AttributionBench reveal the challenges of automatic attribution evaluation, even for state-of-the-art LLMs. Specifically, our findings show that even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation. A detailed analysis of more than 300 error cases indicates that a majority of failures stem from the model's inability to process nuanced information, and the discrepancy between the information the model has access to and that human annotators do.

翻译：现代生成式搜索引擎通过提供引证证据，提升了大型语言模型（LLM）响应的可靠性。然而，评估答案的归因性——即生成响应中的每个主张是否完全得到所引用证据的支持——仍是一个未解决的问题。这种验证传统上依赖昂贵的人工评估，凸显了对自动归因评估方法的迫切需求。为填补这些方法缺乏标准化基准的空白，我们提出了AttributionBench，一个从多个现有归因数据集中整合而成的综合基准。我们在AttributionBench上的大量实验揭示了自动归因评估的挑战，即使对最先进的LLM而言也是如此。具体而言，我们的研究结果表明，即使在二分类公式下，经过微调的GPT-3.5的宏F1分数也仅能达到约80%。对超过300个错误案例的详细分析表明，大多数失败源于模型处理细微信息的能力不足，以及模型可获取信息与人工标注员所掌握信息之间的差异。