Data lakes are massive repositories of raw and heterogeneous data, designed to meet the requirements of modern data storage. Nonetheless, this same philosophy increases the complexity of performing discovery tasks to find relevant data for subsequent processing. As a response to these growing challenges, we present FREYJA, a modern data discovery system capable of effectively exploring data lakes, aimed at finding candidates to perform joins and increase the number of attributes for downstream tasks. More precisely, we want to compute rankings that sort potential joins by their relevance. Modern mechanisms apply advanced table representation learning (TRL) techniques to yield accurate joins. Yet, this incurs high computational costs when dealing with elevated volumes of data. In contrast to the state-of-the-art, we adopt a novel notion of join quality tailored to data lakes, which leverages syntactic measurements while achieving accuracy comparable to that of TRL approaches. To obtain this metric in a scalable manner we train a general purpose predictive model. Predictions are based, rather than on large-scale datasets, on data profiles, succinct representations that capture the underlying characteristics of the data. Our experiments show that our system, FREYJA, matches the results of the state-of-the-art whilst reducing the execution times by several orders of magnitude.

翻译：数据湖是存储原始异构数据的大规模存储库，旨在满足现代数据存储的需求。然而，这种设计理念增加了数据发现任务的复杂性，使得寻找后续处理所需的相关数据变得困难。为应对这些日益严峻的挑战，我们提出了FREYJA——一个能够有效探索数据湖的现代数据发现系统，旨在寻找可执行连接操作的候选数据集，从而为下游任务增加属性数量。更具体地说，我们希望计算能按相关性排序潜在连接的排名列表。现有先进机制通过应用先进的表表示学习（TRL）技术来生成精确连接，但在处理海量数据时会产生高昂的计算成本。与现有技术不同，我们采用了一种专为数据湖设计的新型连接质量度量方法，该方法利用句法测量指标，同时达到与TRL方法相当的准确性。为以可扩展方式获取该度量值，我们训练了一个通用预测模型。该模型的预测并非基于大规模数据集，而是基于数据画像——即捕捉数据底层特征的简洁表示。实验表明，我们的FREYJA系统在保持与现有先进技术相当结果的同时，将执行时间降低了数个数量级。