In recent years there have been considerable advances in pre-trained language models, where non-English language versions have also been made available. Due to their increasing use, many lightweight versions of these models (with reduced parameters) have also been released to speed up training and inference times. However, versions of these lighter models (e.g., ALBERT, DistilBERT) for languages other than English are still scarce. In this paper we present ALBETO and DistilBETO, which are versions of ALBERT and DistilBERT pre-trained exclusively on Spanish corpora. We train several versions of ALBETO ranging from 5M to 223M parameters and one of DistilBETO with 67M parameters. We evaluate our models in the GLUES benchmark that includes various natural language understanding tasks in Spanish. The results show that our lightweight models achieve competitive results to those of BETO (Spanish-BERT) despite having fewer parameters. More specifically, our larger ALBETO model outperforms all other models on the MLDoc, PAWS-X, XNLI, MLQA, SQAC and XQuAD datasets. However, BETO remains unbeaten for POS and NER. As a further contribution, all models are publicly available to the community for future research.

翻译：近年来，预训练语言模型取得了显著进展，非英语版本的语言模型也已相继推出。由于这些模型的应用日益广泛，许多参数更少的轻量级版本也相继发布，以加速训练和推理过程。然而，针对英语以外语言的轻量级模型版本（如ALBERT、DistilBERT）仍然稀缺。本文提出了ALBETO和DistilBETO，这两种模型分别是基于西班牙语语料库专门预训练的ALBERT和DistilBERT版本。我们训练了多个参数规模从5M到223M的ALBETO版本，以及一个参数规模为67M的DistilBETO版本。我们在包含多种西班牙语自然语言理解任务的GLUES基准上对模型进行了评估。结果表明，尽管我们的轻量级模型参数更少，但仍取得了与BETO（西班牙语BERT）相媲美的竞争性结果。具体而言，我们的较大规模ALBETO模型在MLDoc、PAWS-X、XNLI、MLQA、SQAC和XQuAD数据集上均优于其他所有模型。然而，在词性标注和命名实体识别任务上，BETO仍保持领先地位。作为额外贡献，所有模型均已向社区公开，供未来研究使用。