The large language model (LLM) community focuses almost exclusively on decoder-only language models, since they are easier to use for text generation. However, a large subset of the community still uses encoder-only models for tasks such as classification or retrieval. Previous work has attempted to compare these architectures, but is forced to make comparisons with models that have different numbers of parameters, training techniques, and datasets. We introduce the SOTA open-data Ettin suite of models: paired encoder-only and decoder-only models ranging from 17 million parameters to 1 billion, trained on up to 2 trillion tokens. Using the same recipe for both encoder-only and decoder-only models produces SOTA recipes in both categories for their respective sizes, beating ModernBERT as an encoder and Llama 3.2 and SmolLM2 as decoders. Like previous work, we find that encoder-only models excel at classification and retrieval tasks while decoders excel at generative tasks. However, we show that adapting a decoder model to encoder tasks (and vice versa) through continued training is subpar compared to using only the reverse objective (i.e. a 400M encoder outperforms a 1B decoder on MNLI, and vice versa for generative tasks). We open-source all artifacts of this study including training data, training order segmented by checkpoint, and 200+ checkpoints to allow future work to analyze or extend all aspects of training.

翻译：大型语言模型（LLM）领域几乎完全专注于仅解码器语言模型，因为它们在文本生成任务中更易于使用。然而，该领域仍有大量研究者使用仅编码器模型处理分类或检索等任务。先前的研究尝试比较这些架构，但被迫在不同参数量、训练技术和数据集的模型之间进行对比。我们推出了SOTA开源数据模型套件Ettin：包含从1700万到10亿参数量的配对仅编码器与仅解码器模型，训练数据规模高达2万亿词元。对仅编码器与仅解码器模型采用相同的训练方案，在各自规模类别中均产生了SOTA方案——在编码器方面超越了ModernBERT，在解码器方面超越了Llama 3.2与SmolLM2。与先前研究一致，我们发现仅编码器模型在分类和检索任务中表现优异，而解码器在生成任务中更具优势。然而，我们证明通过持续训练使解码器适应编码器任务（反之亦然）的效果，远不及直接使用反向目标模型（例如4亿参数的编码器在MNLI任务上优于10亿参数的解码器，而在生成任务中情况则相反）。本研究的所有成果均已开源，包括训练数据、按检查点划分的训练顺序以及200多个检查点，以便后续研究能对训练全过程进行分析或扩展。