Large Language Models (LLMs) have revolutionized natural language processing and demonstrated impressive capabilities in various tasks. Unfortunately, they are prone to hallucinations, where the model exposes incorrect or false information in its responses, which renders diligent evaluation approaches mandatory. While LLM performance in specific knowledge fields is often evaluated based on question and answer (Q&A) datasets, such evaluations usually report only a single accuracy number for the dataset, which often covers an entire field. This field-based evaluation, is problematic with respect to transparency and model improvement. A stratified evaluation could instead reveal subfields, where hallucinations are more likely to occur and thus help to better assess LLMs' risks and guide their further development. To support such stratified evaluations, we propose LLMMaps as a novel visualization technique that enables users to evaluate LLMs' performance with respect to Q&A datasets. LLMMaps provide detailed insights into LLMs' knowledge capabilities in different subfields, by transforming Q&A datasets as well as LLM responses into an internal knowledge structure. An extension for comparative visualization furthermore, allows for the detailed comparison of multiple LLMs. To assess LLMMaps we use them to conduct a comparative analysis of several state-of-the-art LLMs, such as BLOOM, GPT-2, GPT-3, ChatGPT and LLaMa-13B, as well as two qualitative user evaluations. All necessary source code and data for generating LLMMaps to be used in scientific publications and elsewhere is available on GitHub: https://github.com/viscom-ulm/LLMMaps

翻译：大型语言模型（LLMs）已革新自然语言处理，并在各类任务中展现出令人瞩目的能力。然而，它们容易产生幻觉，即在响应中暴露错误或虚假信息，这迫使我们必须采取严谨的评估方法。尽管特定知识领域的LLM性能常基于问答（Q&A）数据集进行评估，但这种评估通常仅报告数据集（常覆盖整个领域）的单一准确率数值。这种基于领域的评估在透明度和模型改进方面存在问题。分层评估则能揭示更可能发生幻觉的子领域，从而有助于更好地评估LLM风险并指导其进一步发展。为支持此类分层评估，我们提出LLMMaps作为新型可视化技术，使用户能够基于Q&A数据集评估LLM的性能。LLMMaps通过将Q&A数据集及LLM响应转化为内部知识结构，提供关于LLM在不同子领域中知识能力的详细见解。此外，其比较可视化扩展功能允许对多个LLM进行细致对比。为评估LLMMaps，我们将其用于对BLOOM、GPT-2、GPT-3、ChatGPT和LLaMa-13B等前沿LLM进行对比分析，并开展了两项定性用户评估。生成LLMMaps（用于科学出版物及其他场景）所需的所有源代码和数据均可在GitHub上获取：https://github.com/viscom-ulm/LLMMaps