Classification tasks require annotated data, which can often be expensive, time-consuming, or even unfeasible to collect. This is the case of the medical domain, where large datasets often have few annotated examples. To address this, we propose DecSelfMask (Decoder Self-learning by Masking), an approach to enhance decoder-only performance on classification tasks. We build on common self-learning approaches by leveraging a model to create training examples from unlabeled data to propose a novel relevance-guided masking strategy. We use relevance attribution methods to determine what portions of unannotated texts are relevant for a task. We then create self-supervised training examples by masking out those portions, training the model to reconstruct them via next-token-prediction. We hypothesize that those examples convey knowledge about the structure and semantics of unannotated data that can be useful for downstream performance. We test our approach on 136 tasks from a collection of 1.9M clinical notes from an Italian hospital. We quantify DecSelfMask's impact on downstream tasks on 5 models of different scales and families, including a probing analysis. Experiments show consistent gains, outperforming standard supervised fine-tuning approaches (+19.9 points in Macro F1), synthetic label generation (+12.5), and continual pretraining (+6.3), as well as common baselines.

翻译：分类任务需要标注数据，而标注数据的获取往往成本高昂、耗时漫长甚至难以实现。这一问题在医学领域尤为突出，该领域的大规模数据集中标注样本通常很少。为此，我们提出DecSelfMask（解码器自掩码学习）方法，旨在提升仅含解码器模型在分类任务中的性能。我们在常见自学习方法的基础上，通过利用模型从无标注数据中生成训练样本，提出了一种新颖的相关性引导掩码策略。我们使用相关性归因方法确定无标注文本中与任务相关的部分，然后通过掩码这些部分创建自监督训练样本，训练模型通过下一词预测任务对其进行重构。我们假设这些样本蕴含无标注数据结构与语义的知识，有助于提升下游任务性能。我们在包含190万份意大利医院临床记录的136个任务上验证该方法，并针对5种不同规模与架构的模型（含探针分析）量化DecSelfMask对下游任务的影响。实验表明，该方法持续优于标准监督微调（Macro F1提升19.9个百分点）、合成标签生成（提升12.5）、持续预训练（提升6.3）及常用基线方法。