The field of relation extraction (RE) is experiencing a notable shift towards generative relation extraction (GRE), leveraging the capabilities of large language models (LLMs). However, we discovered that traditional relation extraction (RE) metrics like precision and recall fall short in evaluating GRE methods. This shortfall arises because these metrics rely on exact matching with human-annotated reference relations, while GRE methods often produce diverse and semantically accurate relations that differ from the references. To fill this gap, we introduce GenRES for a multi-dimensional assessment in terms of the topic similarity, uniqueness, granularity, factualness, and completeness of the GRE results. With GenRES, we empirically identified that (1) precision/recall fails to justify the performance of GRE methods; (2) human-annotated referential relations can be incomplete; (3) prompting LLMs with a fixed set of relations or entities can cause hallucinations. Next, we conducted a human evaluation of GRE methods that shows GenRES is consistent with human preferences for RE quality. Last, we made a comprehensive evaluation of fourteen leading LLMs using GenRES across document, bag, and sentence level RE datasets, respectively, to set the benchmark for future research in GRE

翻译：关系抽取（RE）领域正经历向生成式关系抽取（GRE）的显著转变，这得益于大语言模型（LLM）的能力。然而，我们发现传统关系抽取指标（如精确率和召回率）在评估GRE方法时存在不足。这种不足源于这些指标依赖与人工标注参考关系的精确匹配，而GRE方法常产生多样且语义准确却不同于参考关系的结果。为弥补这一空白，我们提出GenRES，从主题相似性、独特性、粒度、事实性和完整性五个维度对GRE结果进行多维度评估。借助GenRES，我们实证发现：（1）精确率/召回率无法合理评估GRE方法性能；（2）人工标注的参考关系可能存在不完整性；（3）使用固定关系集或实体集提示LLM会导致幻觉。随后，我们对GRE方法开展人工评估，证明GenRES与人类对RE质量的偏好一致。最终，我们利用GenRES在文档级、袋级和句子级RE数据集上对十四种主流LLM进行全面评估，从而为未来GRE研究建立基准。