We investigate to which extent open large language models (LLMs) can generate coherent and relevant text from structured data. To prevent bias from benchmarks leaked into LLM training data, we collect Quintd-1: an ad-hoc benchmark for five data-to-text (D2T) generation tasks, consisting of structured data records in standard formats gathered from public APIs. We leverage reference-free evaluation metrics and LLMs' in-context learning capabilities, allowing us to test the models with no human-written references. Our evaluation focuses on annotating semantic accuracy errors on token-level, combining human annotators and a metric based on GPT-4. Our systematic examination of the models' behavior across domains and tasks suggests that state-of-the-art open LLMs with 7B parameters can generate fluent and coherent text from various standard data formats in zero-shot settings. However, we also show that semantic accuracy of the outputs remains a major issue: on our benchmark, 80% of outputs of open LLMs contain a semantic error according to human annotators (91% according to GPT-4). Our code, data, and model outputs are available at https://d2t-llm.github.io.

翻译：我们研究了开放大型语言模型（LLMs）在从结构化数据生成连贯且相关文本方面的能力。为避免基准测试数据泄露到LLM训练数据中造成的偏差，我们构建了Quintd-1：一个面向五项数据到文本（D2T）生成任务的特设基准，其数据来源于公共API获取的标准格式结构化记录。我们采用无参考评估指标及LLM的上下文学习能力，从而无需人工撰写参考文本即可测试模型。评估聚焦于词元级别的语义准确性错误标注，结合了人工标注与基于GPT-4的度量方法。通过对模型跨领域和跨任务行为的系统分析，我们发现具有70亿参数的最先进开放LLM能在零样本设置下，从多种标准数据格式生成流畅连贯的文本。然而，我们也揭示输出文本的语义准确性仍是主要挑战：在本基准中，根据人工标注者判断，80%的开放LLM输出存在语义错误（基于GPT-4的评估则高达91%）。我们的代码、数据和模型输出可参见 https://d2t-llm.github.io。