Transformer-based language models have achieved significant success in various domains. However, the data-intensive nature of the transformer architecture requires much labeled data, which is challenging in low-resource scenarios (i.e., few-shot learning (FSL)). The main challenge of FSL is the difficulty of training robust models on small amounts of samples, which frequently leads to overfitting. Here we present Mask-BERT, a simple and modular framework to help BERT-based architectures tackle FSL. The proposed approach fundamentally differs from existing FSL strategies such as prompt tuning and meta-learning. The core idea is to selectively apply masks on text inputs and filter out irrelevant information, which guides the model to focus on discriminative tokens that influence prediction results. In addition, to make the text representations from different categories more separable and the text representations from the same category more compact, we introduce a contrastive learning loss function. Experimental results on public-domain benchmark datasets demonstrate the effectiveness of Mask-BERT.

翻译：基于Transformer的语言模型在多个领域取得了显著成功。然而，Transformer架构对数据的高需求导致其需要大量标注数据，这在低资源场景（即小样本学习）中面临挑战。小样本学习的主要挑战在于难以在少量样本上训练出鲁棒模型，这常导致过拟合问题。本文提出Mask-BERT，一个简洁且模块化的框架，用于帮助基于BERT的架构应对小样本学习。该方法与现有的提示调优、元学习等小样本学习策略存在本质区别。其核心思想是对文本输入选择性应用掩码以过滤无关信息，从而引导模型聚焦于影响预测结果的区分性标记。此外，为增强不同类别文本表征的可分离性并提升同类文本表征的紧致性，我们引入了对比学习损失函数。在公开基准数据集上的实验结果验证了Mask-BERT的有效性。