Crawling national top-level domains has proven to be highly effective for collecting texts in less-resourced languages. This approach has been recently used for South Slavic languages and resulted in the largest general corpora for this language group: the CLASSLA-web 1.0 corpora. Building on this success, we established a continuous crawling infrastructure for iterative national top-level domain crawling across South Slavic and related webs. We present the first outcome of this crawling infrastructure - the CLASSLA-web 2.0 corpus collection, with substantially larger web corpora containing 17.0 billion words in 38.1 million texts in seven languages: Bosnian, Bulgarian, Croatian, Macedonian, Montenegrin, Serbian, and Slovenian. In addition to genre categories, the new version is also automatically annotated with topic labels. Comparing CLASSLA-web 2.0 with its predecessor reveals that only one-fifth of the texts overlap, showing that re-crawling after just two years yields largely new content. However, while the new web crawls bring growing gains, we also notice growing pains - a manual inspection of top domains reveals a visible degradation of web content, as machine-generated sites now contribute a significant portion of texts.
翻译:对国家顶级域名进行网络爬取已被证明是收集资源稀缺语言文本的高效方法。该方法近期被应用于南斯拉夫语族,并为此语系构建了规模最大的通用语料库:CLASSLA-web 1.0系列语料库。基于此成功实践,我们建立了针对南斯拉夫语族及相关网络的迭代式国家顶级域名持续爬取架构。本文展示了该爬取架构的首个成果——CLASSLA-web 2.0语料集,其网络语料规模显著扩大,涵盖波斯尼亚语、保加利亚语、克罗地亚语、马其顿语、黑山语、塞尔维亚语和斯洛文尼亚语等七种语言,包含3810万篇文本共计170亿词。除体裁分类外,新版语料库还通过自动标注增加了主题标签。通过对比CLASSLA-web 2.0与其前代版本发现,仅有五分之一的文本存在重叠,这表明仅间隔两年的重新爬取即可获得大量新内容。然而,尽管新版网络爬取带来了持续增长的数据收益,我们也注意到随之加剧的阵痛——对顶级域名的人工核查显示网络内容质量出现明显退化,当前机器生成站点已贡献了相当比例的文本。