The recent introduction of Transformers language representation models allowed great improvements in many natural language processing (NLP) tasks. However, if on one hand the performances achieved by this kind of architectures are surprising, on the other their usability is limited by the high number of parameters which constitute their network, resulting in high computational and memory demands. In this work we present BERTino, a DistilBERT model which proposes to be the first lightweight alternative to the BERT architecture specific for the Italian language. We evaluated BERTino on the Italian ISDT, Italian ParTUT, Italian WikiNER and multiclass classification tasks, obtaining F1 scores comparable to those obtained by a BERTBASE with a remarkable improvement in training and inference speed.

翻译：近年来，Transformer语言表征模型的引入使诸多自然语言处理任务取得了显著进展。然而，这类架构虽在性能上表现惊艳，其网络包含的大量参数却导致计算与内存需求过高，限制了其实用性。本文提出BERTino——一个面向意大利语的DistilBERT模型，旨在成为首个专为意大利语设计的轻量级BERT架构替代方案。我们分别在意大利语ISDT、意大利语ParTUT、意大利语WikiNER数据集及多类别分类任务上对BERTino进行评估，结果显示其F1分数与BERTBASE相当，同时训练和推理速度显著提升。