Large-scale transformer-based models like the Bidirectional Encoder Representations from Transformers (BERT) are widely used for Natural Language Processing (NLP) applications, wherein these models are initially pre-trained with a large corpus with millions of parameters and then fine-tuned for a downstream NLP task. One of the major limitations of these large-scale models is that they cannot be deployed on resource-constrained devices due to their large model size and increased inference latency. In order to overcome these limitations, such large-scale models can be converted to an optimized FlatBuffer format, tailored for deployment on resource-constrained edge devices. Herein, we evaluate the performance of such FlatBuffer transformed MobileBERT models on three different edge devices, fine-tuned for Reputation analysis of English language tweets in the RepLab 2013 dataset. In addition, this study encompassed an evaluation of the deployed models, wherein their latency, performance, and resource efficiency were meticulously assessed. Our experiment results show that, compared to the original BERT large model, the converted and quantized MobileBERT models have 160$\times$ smaller footprints for a 4.1% drop in accuracy while analyzing at least one tweet per second on edge devices. Furthermore, our study highlights the privacy-preserving aspect of TinyML systems as all data is processed locally within a serverless environment.

翻译：大规模基于Transformer的模型，如BERT（来自Transformer的双向编码器表示），广泛用于自然语言处理（NLP）应用。这些模型首先通过包含数百万参数的大规模语料库进行预训练，随后针对下游NLP任务进行微调。这类大规模模型的主要局限之一在于，由于其庞大的模型尺寸和增加的推理延迟，无法部署在资源受限的设备上。为了克服这些局限，可将此类大规模模型转换为优化的FlatBuffer格式，专为部署在资源受限的边缘设备而设计。本文评估了三种不同边缘设备上经FlatBuffer转换的MobileBERT模型的性能，这些模型针对RepLab 2013数据集中的英文推文声誉分析任务进行了微调。此外，本研究还包含对部署模型的评估，对其延迟、性能和资源效率进行了细致分析。实验结果表明，与原始BERT大规模模型相比，转换并量化的MobileBERT模型在边缘设备上以每秒至少分析一条推文的速度运行时，大小缩小了160倍，而准确率仅下降4.1%。此外，本研究强调了TinyML系统在隐私保护方面的优势，因为所有数据均在无服务器环境中进行本地处理。