Recent regulatory initiatives like the European AI Act and relevant voices in the Machine Learning (ML) community stress the need to describe datasets along several key dimensions for trustworthy AI, such as the provenance processes and social concerns. However, this information is typically presented as unstructured text in accompanying documentation, hampering their automated analysis and processing. In this work, we explore using large language models (LLM) and a set of prompting strategies to automatically extract these dimensions from documents and enrich the dataset description with them. Our approach could aid data publishers and practitioners in creating machine-readable documentation to improve the discoverability of their datasets, assess their compliance with current AI regulations, and improve the overall quality of ML models trained on them. In this paper, we evaluate the approach on 12 scientific dataset papers published in two scientific journals (Nature's Scientific Data and Elsevier's Data in Brief) using two different LLMs (GPT3.5 and Flan-UL2). Results show good accuracy with our prompt extraction strategies. Concrete results vary depending on the dimensions, but overall, GPT3.5 shows slightly better accuracy (81,21%) than FLAN-UL2 (69,13%) although it is more prone to hallucinations. We have released an open-source tool implementing our approach and a replication package, including the experiments' code and results, in an open-source repository.

翻译：近期的监管举措（如欧盟《人工智能法案》）以及机器学习社区的相关声音强调，需在可信人工智能框架下从多个关键维度（如溯源过程与社会关切）描述数据集。然而，这些信息通常以非结构化文本形式呈现于配套文档中，阻碍了自动化分析与处理。本研究探索利用大型语言模型（LLM）及一组提示策略，从文档中自动提取这些维度，并以此丰富数据集描述。我们的方法可帮助数据发布者与实践者创建机器可读文档，提升数据集的可发现性、评估其对现行人工智能法规的合规性，并改进基于这些数据集训练的机器学习模型的整体质量。本文在发表于两本科学期刊（Nature旗下《Scientific Data》与Elsevier旗下《Data in Brief》）的12篇科学数据集论文中，使用两种不同的LLM（GPT3.5与Flan-UL2）评估了该方法。实验结果表明，我们的提示提取策略具有良好的准确性。具体结果因维度而异，但总体而言，GPT3.5的准确率（81.21%）略高于FLAN-UL2（69.13%），不过GPT3.5更易产生幻觉。我们已在开源仓库中发布了实现该方法的开源工具及包含实验代码与结果的复现包。