Large Language Models (LLMs) have revolutionized natural language processing and demonstrated impressive capabilities in various tasks. Unfortunately, they are prone to hallucinations, where the model exposes incorrect or false information in its responses, which renders diligent evaluation approaches mandatory. While LLM performance in specific knowledge fields is often evaluated based on question and answer (Q&A) datasets, such evaluations usually report only a single accuracy number for the entire field, a procedure which is problematic with respect to transparency and model improvement. A stratified evaluation could instead reveal subfields, where hallucinations are more likely to occur and thus help to better assess LLMs' risks and guide their further development. To support such stratified evaluations, we propose LLMMaps as a novel visualization technique that enables users to evaluate LLMs' performance with respect to Q&A datasets. LLMMaps provide detailed insights into LLMs' knowledge capabilities in different subfields, by transforming Q&A datasets as well as LLM responses into our internal knowledge structure. An extension for comparative visualization furthermore, allows for the detailed comparison of multiple LLMs. To assess LLMMaps we use them to conduct a comparative analysis of several state-of-the-art LLMs, such as BLOOM, GPT-2, GPT-3, ChatGPT and LLaMa-13B, as well as two qualitative user evaluations. All necessary source code and data for generating LLMMaps to be used in scientific publications and elsewhere will be available on GitHub.

翻译：大型语言模型（LLMs）革新了自然语言处理，并在各项任务中展现出令人印象深刻的能力。然而，它们容易产生幻觉，即模型在其回复中暴露出错误或虚假信息，这迫使必须采用严谨的评估方法。尽管特定知识领域的LLM性能通常基于问答（Q&A）数据集进行评估，但这种评估通常仅报告整个领域的单一准确率数值，这一流程在透明度和模型改进方面存在问题。分层评估能够揭示更易出现幻觉的子领域，从而有助于更好地评估LLM的风险并指导其进一步开发。为支持此类分层评估，我们提出LLMMaps作为一种新颖的可视化技术，使用户能够针对Q&A数据集评估LLM的性能。LLMMaps通过将Q&A数据集以及LLM的回复转换为我们内部的知识结构，提供了对不同子领域中LLM知识能力的细致洞察。此外，一种支持对比可视化的扩展功能，能实现对多个LLM的详细比较。为评估LLMMaps，我们利用其对多个最先进的LLM（如BLOOM、GPT-2、GPT-3、ChatGPT和LLaMa-13B）进行了比较分析，并开展了两次定性用户评估。用于在科学出版物及其他场合生成LLMMaps所需的所有源代码和数据将在GitHub上公开。