Somali is a Cushitic language of the Horn of Africa with ~25 million speakers, yet no documented dedicated Somali pretraining corpus with a companion tokenizer and language-identification benchmark has been publicly released. Existing Somali text appears either inside multilingual distributions (HPLT v2, CC100, MADLAD-400, OSCAR, mC4) or in small, undocumented Somali-only uploads on Hugging Face. We introduce SomaliWeb v1, a quality-filtered Somali corpus of 819,322 documents (~303M tokens) built from three upstream sources (HPLT v2, CC100, Somali Wikipedia) through a six-stage reproducible pipeline. We release (i) the corpus, (ii) a matched BPE-16K tokenizer, and (iii) the first public side-by-side Somali benchmark of three production language identifiers. Our measurements reveal concrete quality defects in existing distributions: HPLT v2's "cleaned" Somali release retains 17.3% byte-exact duplicates, 56.1% of its documents contain fixable mojibake, and 10.7% of its byte-unique documents are near-duplicates at Jaccard tau=0.80. Our BPE-16K tokenizer emits 40.2% fewer tokens than GPT-4's cl100k_base on FLORES-200 Somali devtest as a tokenizer-level measurement; downstream language-model perplexity comparisons are deferred to a follow-up release.

翻译：索马里语是非洲之角的一种库希特语族语言，约有2500万使用者。然而，迄今为止尚未公开推出任何经过专门设计的索马里语预训练语料库及其配套分词器和语言识别基准测试。现有的索马里语文本要么出现在多语言分布（HPLT v2、CC100、MADLAD-400、OSCAR、mC4）中，要么以小型且未文档化的仅索马里语形式上传至Hugging Face。我们推出了SomaliWeb v1，一个质量过滤的索马里语语料库，包含819,322篇文档（约3.03亿个token），通过一个六阶段可复现流水线从三个上游来源（HPLT v2、CC100、索马里语维基百科）构建而成。我们发布了（i）该语料库，（ii）配套的BPE-16K分词器，以及（iii）首个公开的包含三种生产级语言识别器的索马里语并排基准测试。我们的测量揭示了现有分布中的具体质量缺陷：HPLT v2的"清洗后"索马里语版本仍含有17.3%的字节级精确重复文档，56.1%的文档包含可修复的乱码文本，10.7%的字节唯一文档在Jaccard相似度tau=0.80条件下为近似重复文档。我们的BPE-16K分词器在FLORES-200索马里语开发测试集上作为分词器级测量，产生的token比GPT-4的cl100k_base少40.2%；下游语言模型困惑度比较的结果留待后续发布。