Translating natural language to visualization (NL2VIS) has shown great promise for visual data analysis, but it remains a challenging task that requires multiple low-level implementations, such as natural language processing and visualization design. Recent advancements in pre-trained large language models (LLMs) are opening new avenues for generating visualizations from natural language. However, the lack of a comprehensive and reliable benchmark hinders our understanding of LLMs' capabilities in visualization generation. In this paper, we address this gap by proposing a new NL2VIS benchmark called VisEval. Firstly, we introduce a high-quality and large-scale dataset. This dataset includes 2,524 representative queries covering 146 databases, paired with accurately labeled ground truths. Secondly, we advocate for a comprehensive automated evaluation methodology covering multiple dimensions, including validity, legality, and readability. By systematically scanning for potential issues with a number of heterogeneous checkers, VisEval provides reliable and trustworthy evaluation outcomes. We run VisEval on a series of state-of-the-art LLMs. Our evaluation reveals prevalent challenges and delivers essential insights for future advancements.

翻译：将自然语言转换为可视化（NL2VIS）在视觉数据分析领域展现出巨大潜力，但其仍是一项需要自然语言处理和可视化设计等多层次实现的复杂任务。预训练大语言模型（LLMs）的最新进展为从自然语言生成可视化开辟了新途径。然而，由于缺乏全面可靠的基准测试体系，我们对于LLMs在可视化生成方面的能力认知仍存在局限。本文通过提出名为VisEval的新型NL2VIS基准测试来解决这一空白。首先，我们构建了高质量大规模数据集，该数据集包含覆盖146个数据库的2,524个代表性查询，并配有精确标注的真实值。其次，我们倡导涵盖有效性、合法性与可读性等多维度的自动化评估方法体系。通过采用异构检查器系统扫描潜在问题，VisEval能够提供可靠且可信的评估结果。我们在系列前沿LLMs上运行VisEval进行评估，实验揭示了当前普遍存在的挑战，并为未来发展提供了关键见解。