We investigated whether large language models (LLMs) can develop data validation tests. We considered 96 conditions each for both GPT-3.5 and GPT-4, examining different prompt scenarios, learning modes, temperature settings, and roles. The prompt scenarios were: 1) Asking for expectations, 2) Asking for expectations with a given context, 3) Asking for expectations after requesting a data simulation, and 4) Asking for expectations with a provided data sample. The learning modes were: 1) zero-shot, 2) one-shot, and 3) few-shot learning. We also tested four temperature settings: 0, 0.4, 0.6, and 1. And the two distinct roles were: 1) helpful assistant, 2) expert data scientist. To gauge consistency, every setup was tested five times. The LLM-generated responses were benchmarked against a gold standard data validation suite, created by an experienced data scientist knowledgeable about the data in question. We find there are considerable returns to the use of few-shot learning, and that the more explicit the data setting can be the better, to a point. The best LLM configurations complement, rather than substitute, the gold standard results. This study underscores the value LLMs can bring to the data cleaning and preparation stages of the data science workflow, but highlights that they need considerable evaluation by experienced analysts.

翻译：本研究探究了大语言模型（LLMs）能否开发数据验证测试。针对GPT-3.5和GPT-4，我们分别设置了96组条件，考察不同提示场景、学习模式、温度设置和角色设定对结果的影响。提示场景包括：1）直接提出期望要求；2）在给定上下文下提出期望；3）请求数据模拟后提出期望；4）提供数据样本后提出期望。学习模式分为：1）零样本学习；2）单样本学习；3）少样本学习。我们测试了四个温度参数：0、0.4、0.6和1，以及两种角色设定：1）辅助助手；2）数据专家。为评估一致性，每种配置均重复测试五次。将LLM生成的响应与由资深数据科学家（熟悉该数据）创建的标准数据验证套件进行基准对比。研究发现：少样本学习可显著提升效果，且数据设定越明确（在一定范围内）效果越好。最优LLM配置能补充而非替代标准验证结果。本研究凸显了LLM在数据科学工作流的数据清洗与准备阶段的价值，但强调仍需经验丰富的分析师对其进行充分评估。