Different entities with the same name can be difficult to distinguish. Handling confusing entity mentions is a crucial skill for language models (LMs). For example, given the question "Where was Michael Jordan educated?" and a set of documents discussing different people named Michael Jordan, can LMs distinguish entity mentions to generate a cohesive answer to the question? To test this ability, we introduce a new benchmark, AmbigDocs. By leveraging Wikipedia's disambiguation pages, we identify a set of documents, belonging to different entities who share an ambiguous name. From these documents, we generate questions containing an ambiguous name and their corresponding sets of answers. Our analysis reveals that current state-of-the-art models often yield ambiguous answers or incorrectly merge information belonging to different entities. We establish an ontology categorizing four types of incomplete answers and automatic evaluation metrics to identify such categories. We lay the foundation for future work on reasoning across multiple documents with ambiguous entities.

翻译：同名不同实体常难以区分。处理混淆的实体提及是语言模型（LM）的关键技能。例如，面对“迈克尔·乔丹在哪里接受教育？”这一问题以及一组讨论名为迈克尔·乔丹的不同人物的文档时，LM能否区分实体提及以生成连贯的回答？为检验这一能力，我们提出了新基准AmbigDocs。通过利用维基百科消歧义页面，我们识别了属于共享歧义名称的不同实体的一组文档。基于这些文档，我们生成了包含歧义名称的问题及其对应的答案集。分析表明，当前最先进的模型常给出歧义答案，或错误地合并不同实体的信息。我们建立了包含四种不完整答案类型的本体以及用于识别这些类型的自动评估指标。这为未来针对含歧义实体的多文档推理研究奠定了基础。