Large Language Models (LLMs) have revolutionized natural language processing and demonstrated impressive capabilities in various tasks. Unfortunately, they are prone to hallucinations, where the model exposes incorrect or false information in its responses, which renders diligent evaluation approaches mandatory. While LLM performance in specific knowledge fields is often evaluated based on question and answer (Q&A) datasets, such evaluations usually report only a single accuracy number for the entire field, a procedure which is problematic with respect to transparency and model improvement. A stratified evaluation could instead reveal subfields, where hallucinations are more likely to occur and thus help to better assess LLMs' risks and guide their further development. To support such stratified evaluations, we propose LLMMaps as a novel visualization technique that enables users to evaluate LLMs' performance with respect to Q&A datasets. LLMMaps provide detailed insights into LLMs' knowledge capabilities in different subfields, by transforming Q&A datasets as well as LLM responses into our internal knowledge structure. An extension for comparative visualization furthermore, allows for the detailed comparison of multiple LLMs. To assess LLMMaps we use them to conduct a comparative analysis of several state-of-the-art LLMs, such as BLOOM, GPT-2, GPT-3, ChatGPT and LLaMa-13B, as well as two qualitative user evaluations. All necessary source code and data for generating LLMMaps to be used in scientific publications and elsewhere will be available on GitHub.

翻译：大语言模型（LLMs）彻底改变了自然语言处理，并在各种任务中展现出令人印象深刻的能力。不幸的是，它们容易产生幻觉，即模型在其响应中暴露错误或不真实信息，这使得必须采用严格的评估方法。虽然特定知识领域的LLM性能通常基于问答（Q&A）数据集进行评估，但这种评估通常只报告整个领域的单一准确率数字，这一过程在透明度和模型改进方面存在问题。分层评估反而能揭示更易发生幻觉的子领域，从而有助于更好地评估LLM的风险并指导其进一步开发。为支持此类分层评估，我们提出LLMMaps作为一种新颖的可视化技术，使用户能够评估LLM在问答数据集上的表现。LLMMaps通过将问答数据集以及LLM的响应转换为我们内部的知识结构，提供LLM在不同子领域中知识能力的详细见解。此外，一个用于比较可视化的扩展功能允许对多个LLM进行详细比较。为评估LLMMaps，我们利用它对若干最先进的LLM（如BLOOM、GPT-2、GPT-3、ChatGPT和LLaMa-13B）进行了比较分析，并进行了两次定性用户评估。用于生成LLMMaps以便在科学出版物及其他场合使用的所有必要源代码和数据将可在GitHub上获取。