When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily switching to Causal Language Modeling (CLM) followed by a short MLM decay improves downstream performance. On biomedical texts with ModernBERT, this CLM detour outperforms MLM baselines trained on identical data and compute across 8 French and 11 English biomedical tasks, by +1.2-2.8pp and +0.3-0.8pp respectively, depending on model size. We investigate the reasons for these gains. We find that CLM's dense supervision impacts low transformer layers (0-7) far more than MLM does. Freezing low layers during CLM eliminates the downstream benefit; freezing mid layers preserves it. The representational changes persist through the MLM decay phase, even when it matches the CLM phase in length, and they scale with model capacity. We release ModernCamemBERT-bio and ModernBERT-bio as state-of-the-art biomedical encoders in Base and Large sizes.

翻译：当将编码器适应新领域时，标准方法是使用掩码语言模型（MLM）继续训练。我们证明，暂时切换至因果语言模型（CLM）后再进行短期的MLM衰减，能够提升下游任务性能。在生物医学文本上使用ModernBERT，这种CLM绕路法在8项法语和11项英语生物医学任务中，均优于在相同数据和算力下训练的MLM基线，分别提升+1.2-2.8个百分点和+0.3-0.8个百分点（依模型规模而定）。我们探究了这些增益的原因。发现CLM的密集监督对低层Transformer层（0-7层）的影响远大于MLM。在CLM期间冻结低层会消除下游收益；冻结中层则保留收益。表征变化会持续至MLM衰减阶段（即使该阶段长度与CLM阶段相当），且随模型容量扩展而增强。我们发布ModernCamemBERT-bio和ModernBERT-bio，作为Base和Large规模的生物医学编码器。