Training deep neural networks (DNNs) under weak supervision has attracted increasing research attention as it can significantly reduce the annotation cost. However, labels from weak supervision can be noisy, and the high capacity of DNNs enables them to easily overfit the label noise, resulting in poor generalization. Recent methods leverage self-training to build noise-resistant models, in which a teacher trained under weak supervision is used to provide highly confident labels for teaching the students. Nevertheless, the teacher derived from such frameworks may have fitted a substantial amount of noise and therefore produce incorrect pseudo-labels with high confidence, leading to severe error propagation. In this work, we propose Meta Self-Refinement (MSR), a noise-resistant learning framework, to effectively combat label noise from weak supervision. Instead of relying on a fixed teacher trained with noisy labels, we encourage the teacher to refine its pseudo-labels. At each training step, MSR performs a meta gradient descent on the current mini-batch to maximize the student performance on a clean validation set. Extensive experimentation on eight NLP benchmarks demonstrates that MSR is robust against label noise in all settings and outperforms state-of-the-art methods by up to 11.4% in accuracy and 9.26% in F1 score.

翻译：在弱监督下训练深度神经网络（DNN）因其能显著降低标注成本而受到越来越多的研究关注。然而，弱监督产生的标签可能包含噪声，且DNN的高容量使其易于过拟合标签噪声，导致泛化性能不佳。近期方法利用自训练构建抗噪声模型，其中在弱监督下训练的教师网络用于提供高置信度标签以指导学生网络。然而，此类框架衍生的教师网络可能已拟合大量噪声，从而以高置信度生成错误伪标签，导致严重误差传播。本文提出元自精炼（MSR），一种抗噪声学习框架，可有效对抗弱监督中的标签噪声。我们不依赖固定教师网络使用带噪标签训练，而是鼓励教师网络精炼其伪标签。在每个训练步骤中，MSR在当前小批量数据上执行元梯度下降，以最大化学生网络在干净验证集上的性能。在八个NLP基准上的大量实验表明，MSR在所有设置中对标签噪声均具有鲁棒性，在准确率上领先最先进方法最高达11.4%，在F1值上领先最高达9.26%。