The automatic generation of visualizations is an old task that, through the years, has shown more and more interest from the research and practitioner communities. Recently, large language models (LLM) have become an interesting option for supporting generative tasks related to visualization, demonstrating initial promising results. At the same time, several pitfalls, like the multiple ways of instructing an LLM to generate the desired result, the different perspectives leading the generation (code-based, image-based, grammar-based), and the presence of hallucinations even for the visualization generation task, make their usage less affordable than expected. Following similar initiatives for benchmarking LLMs, this paper copes with the problem of modeling the evaluation of a generated visualization through an LLM. We propose a theoretical evaluation stack, EvaLLM, that decomposes the evaluation effort in its atomic components, characterizes their nature, and provides an overview of how to implement and interpret them. We also designed and implemented an evaluation platform that provides a benchmarking resource for the visualization generation task. The platform supports automatic and manual scoring conducted by multiple assessors to support a fine-grained and semantic evaluation based on the EvaLLM stack. Two case studies on GPT3.5-turbo with Code Interpreter and Llama2-70-b models show the benefits of EvaLLM and illustrate interesting results on the current state-of-the-art LLM-generated visualizations.

翻译：可视化自动生成是一项历史悠久的任务，多年来已引起研究界与实践界越来越多的关注。近年来，大型语言模型（LLM）已成为支持可视化相关生成任务的新兴选择，并展现出初步的可喜成果。然而，指导LLM生成预期结果的多种方式、主导生成过程的不同视角（基于代码、图像或语法），以及可视化生成任务中出现的幻觉现象等若干问题，使得LLM的实际应用并不像预期那样便捷。借鉴LLM基准测试领域的类似研究，本文旨在解决通过LLM对生成结果进行建模评估的问题。我们提出一个理论性评估栈——EvaLLM，该架构将评估工作分解为原子组件，刻画各组件的特性，并提供实施与解析的整体方案。我们还设计并实现了一个评估平台，为可视化生成任务提供基准测试资源。该平台支持由多名评估者进行的自动与手动评分，以实现基于EvaLLM栈的细粒度语义评估。针对GPT3.5-turbo（含代码解释器）与Llama2-70-b模型的两项案例研究，验证了EvaLLM的优势，并揭示了当前最先进LLM生成可视化结果的相关重要发现。