It is known that LLMs do hallucinate, that is, they return incorrect information as facts. In this paper, we introduce the possibility to study these hallucinations under a structured form: graphs. Hallucinations in this context are incorrect outputs when prompted for well known graphs from the literature (e.g. Karate club, Les Mis\'erables, graph atlas). These hallucinated graphs have the advantage of being much richer than the factual accuracy -- or not -- of a statement; this paper thus argues that such rich hallucinations can be used to characterize the outputs of LLMs. Our first contribution observes the diversity of topological hallucinations from major modern LLMs. Our second contribution is the proposal of a metric for the amplitude of such hallucinations: the Graph Atlas Distance, that is the average graph edit distance from several graphs in the graph atlas set. We compare this metric to the Hallucination Leaderboard, a hallucination rank that leverages 10,000 times more prompts to obtain its ranking.

翻译：众所周知，大型语言模型（LLMs）确实会产生幻觉，即它们会将错误信息作为事实返回。本文提出了一种以结构化形式——图——来研究此类幻觉的可能性。在此语境下，幻觉指的是当提示模型生成文献中知名图结构（例如Karate club、Les Misérables、graph atlas）时产生的错误输出。这些幻觉图相较于单纯判断陈述的事实准确性具有更丰富的内涵；因此，本文认为此类丰富的幻觉可用于表征大型语言模型的输出特性。我们的第一个贡献是观察了主流现代大型语言模型所产生的拓扑幻觉的多样性。我们的第二个贡献是提出了一种衡量此类幻觉幅度的指标：图集距离，即与图集集合中多个图的平均图编辑距离。我们将此指标与幻觉排行榜进行了比较，后者是一个利用超过10,000倍提示量来获取排名的幻觉排序。