Web archives preserve portions of the web, but quantifying their completeness remains challenging. Prior approaches have estimated the coverage of a crawl by either comparing the outcomes of multiple crawlers, or by comparing the results of a single crawl to external ground truth datasets. We propose a method to estimate the absolute coverage of a crawl using only the archive's own longitudinal data, i.e., the data collected by multiple subsequent crawls. Our key insight is that coverage can be estimated from the empirical URL overlaps between subsequent crawls, which are in turn well described by a simple urn process. The parameters of the urn model can then be inferred from longitudinal crawl data using linear regression. Applied to our focused crawl configuration of the German Academic Web, with 15 semi-annual crawls between 2013-2021, we find a coverage of approximately 46 percent of the crawlable URL space for the stable crawl configuration regime. Our method is extremely simple, requires no external ground truth, and generalizes to any longitudinal focused crawl.

翻译：网络档案保存了部分网络内容，但量化其完整性仍具挑战性。以往方法通过比较多个爬虫的输出结果，或比较单次爬虫与外部真实数据集的结果来估计爬虫覆盖率。我们提出一种仅利用档案自身的纵向数据（即多次后续爬虫收集的数据）来估计爬虫绝对覆盖率的方法。关键洞见在于，覆盖率可通过后续爬虫之间的经验性URL重叠来估计，而这一重叠现象可由一个简单的瓮模型很好描述。随后，利用线性回归可从纵向爬虫数据中推断瓮模型的参数。将其应用于我们对德国学术网络的聚焦爬虫配置（2013-2021年间15次半年度爬取），发现在稳定爬虫配置模式下，覆盖率约为可爬取URL空间的46%。该方法极其简单，无需外部真实数据，且可推广至任何纵向聚焦爬虫。