We analyze the behaviors of open large language models (LLMs) on the task of data-to-text (D2T) generation, i.e., generating coherent and relevant text from structured data. To avoid the issue of LLM training data contamination with standard benchmarks, we design Quintd - a tool for collecting novel structured data records from public APIs. Using a dataset collected with Quintd and leveraging reference-free evaluation, we analyze model behaviors on five D2T generation tasks. We find that recent open LLMs (Llama2, Mistral, and Zephyr) can generate fluent and coherent text from standard data formats in zero-shot settings. However, we also show that the semantic accuracy of the outputs is a major issue: both according to our GPT-4-based metric and human annotators, more than 80% of the outputs of open LLMs contain a semantic error. We publicly release the code, data, and model outputs.

翻译：我们分析了开放大型语言模型（LLMs）在数据到文本（D2T）生成任务中的行为，即从结构化数据生成连贯且相关的文本。为避免标准基准测试中LLM训练数据污染的问题，我们设计了Quintd——一种从公共API收集新型结构化数据记录的工具。利用通过Quintd收集的数据集并采用无参考评估方法，我们在五项D2T生成任务上分析了模型行为。研究发现，近期的开放LLM（如Llama2、Mistral和Zephyr）能够在零样本设置下从标准数据格式生成流畅连贯的文本。然而，我们也指出输出文本的语义准确性是一个主要问题：根据我们基于GPT-4的评估指标和人工标注结果，开放LLM超过80%的输出存在语义错误。我们已公开发布相关代码、数据和模型输出。