The large-scale digitization of historical archives has created a paradox: "dark data"-digital objects lacking metadata for retrieval. Manual archival description is slow and expensive, limiting discovery and reuse. We propose Vidya, a modular pipeline that orchestrates Large Language Models (LLMs) and FOSS tools to automate semantic enrichment and archival ingestion at scale. Vidya constrains generations using YAML-defined ontologies and Pydantic validation, producing deterministic, structured JSON outputs from probabilistic models. Developed at Laboratory for Digital Humanities and Innovation (LAMUHDI) of the State University of Ponta Grossa (UEPG), Vidya applies Maker principles and open-source practices to enable low-cost deployment in memory institutions using modest hardware. We compare LLM performance and present a cost-benefit analysis showing major gains, reducing processing time from decades to days while complying with NOBRADE and ISAD(G).
翻译:摘要:历史档案的大规模数字化引发了一个悖论:“暗数据”——即缺乏检索元数据的数字对象。传统人工档案描述方式效率低下、成本高昂,限制了数据的发现与复用。我们提出Vidya——一种模块化流水线,通过协调大型语言模型(LLM)与开源工具,实现大规模语义增强与档案摄入自动化。Vidya利用YAML定义的本体论与Pydantic验证约束生成过程,从概率模型中输出确定性的结构化JSON结果。该工具由蓬塔格罗萨州立大学(UEPG)数字人文与创新实验室(LAMUHDI)开发,融合创客原则与开源实践,支持记忆机构在低配硬件上实现低成本部署。通过对比不同LLM性能并进行成本效益分析,研究表明该方法可将处理时间从数十年缩短至数天,同时符合NOBRADE与ISAD(G)标准。