Encoder-only Transformers have advanced along three axes -- architecture, data, and systems -- yielding Pareto gains in accuracy, speed, and memory efficiency. Yet these improvements have not fully transferred to Chinese, where tokenization and morphology differ markedly from English. We introduce Chinese ModernBERT, a from-scratch Chinese encoder that couples: (i) a hardware-aware 32k BPE vocabulary tailored to frequent Chinese affixes/compounds, lowering the embedding budget; (ii) whole-word masking (WWM) with a dynamic masking curriculum (30% -> 15%) to align task difficulty with training progress; (iii) a two-stage pre-training pipeline that extends the native context from 1,024 to 8,192 tokens using RoPE and alternating local/global attention; and (iv) a damped-cosine learning-rate schedule for stable long-horizon optimization. We pre-train on ~1.2T Chinese tokens from CCI3-HQ, CCI4 (Chinese), and Cosmopedia-Chinese. On CLUE, Chinese ModernBERT is competitive with strong Chinese encoders under a unified fine-tuning protocol. Under bf16 it achieves high long-sequence throughput while maintaining strong short-sequence speed, reflecting benefits from budget allocation and attention design. To probe retrieval-oriented quality, we add a small amount of open contrastive data: fine-tuning on SimCLUE (~3M pairs) improves further when adding T2Ranking (~2M), reaching 0.505 (Pearson) / 0.537 (Spearman) on the SimCLUE test set. Under this open-data setting, Chinese ModernBERT surpasses Qwen-0.6B-embedding on SimCLUE, suggesting a clear scaling path for STS with additional curated pairs. We will release tokenizer and weights to facilitate reproducible research.

翻译：仅编码器Transformer模型在架构、数据和系统三个维度上持续演进，实现了精度、速度与内存效率的帕累托改进。然而这些改进尚未完全迁移至中文领域，因为中文在分词与形态学特征上与英文存在显著差异。本文提出Chinese ModernBERT——一个从头训练的中文编码器，其整合了以下关键技术：(i) 针对硬件优化的32k BPE词表，专门适配高频中文词缀/复合词，降低嵌入层计算开销；(ii) 采用动态掩码课程（30%→15%）的全词掩码策略，使任务难度与训练进度相匹配；(iii) 两阶段预训练流程，通过RoPE与局部/全局交替注意力机制将原生上下文长度从1,024扩展至8,192词元；(iv) 阻尼余弦学习率调度策略，确保长周期优化的稳定性。我们在约1.2万亿中文词元（来自CCI3-HQ、CCI4中文语料及Cosmopedia中文版）上进行预训练。在CLUE基准测试中，采用统一微调协议时，Chinese ModernBERT与主流中文编码器性能相当。在bf16精度下，模型在保持优异短序列处理速度的同时，实现了高吞吐量的长序列处理能力，这体现了预算分配与注意力设计的优势。为探究检索导向的性能，我们引入少量开放对比数据：在SimCLUE（约300万对）微调基础上，进一步加入T2Ranking（约200万对）数据，使模型在SimCLUE测试集上的皮尔逊/斯皮尔曼相关系数达到0.505/0.537。在此开放数据设定下，Chinese ModernBERT在SimCLUE上超越Qwen-0.6B-embedding，表明通过增加精标数据对，语义文本相似度任务存在明确的扩展路径。我们将公开分词器与模型权重以促进可复现研究。