For years, adversarial training has been extensively studied in natural language processing (NLP) settings. The main goal is to make models robust so that similar inputs derive in semantically similar outcomes, which is not a trivial problem since there is no objective measure of semantic similarity in language. Previous works use an external pre-trained NLP model to tackle this challenge, introducing an extra training stage with huge memory consumption during training. However, the recent popular approach of contrastive learning in language processing hints a convenient way of obtaining such similarity restrictions. The main advantage of the contrastive learning approach is that it aims for similar data points to be mapped close to each other and further from different ones in the representation space. In this work, we propose adversarial training with contrastive learning (ATCL) to adversarially train a language processing task using the benefits of contrastive learning. The core idea is to make linear perturbations in the embedding space of the input via fast gradient methods (FGM) and train the model to keep the original and perturbed representations close via contrastive learning. In NLP experiments, we applied ATCL to language modeling and neural machine translation tasks. The results show not only an improvement in the quantitative (perplexity and BLEU) scores when compared to the baselines, but ATCL also achieves good qualitative results in the semantic level for both tasks without using a pre-trained model.

翻译：多年来,在自然语言处理(NLP)环境中广泛研究了对抗性培训。主要目标是使模型变得稳健,使类似的投入在语义相似的结果中产生类似的结果,这不是一个小问题,因为没有客观的语言语义相似性衡量标准。以前的工作使用外部培训前的NLP模式来应对这一挑战,引入一个额外的培训阶段,在培训期间大量记忆消耗。然而,最近在语言处理中进行对比学习的流行方法是获得类似限制的方便方式。对比学习方法的主要优点是,它的目标是将相似的数据点定位在彼此之间,从代表空间的不同方面更接近。在这项工作中,我们建议用对比学习(ATCL)进行对抗性培训,以便利用对比学习的好处对语言处理任务进行对抗性培训。核心思想是通过快速梯度方法(FGM)在投入的嵌入空间中进行线性扰动,并且通过对比学习来保持原始和扰动式的表达方式密切。在NLP实验中,我们用对比性学习(ATCL) 将ACTL的对比性培训结果用于B的升级,在测试中,而没有将AFL的升级为B的排序基准任务。在测试中,只是在测试中,在测试中,在测试中,在测试后,在测试后,在测试后,在测试后,在测试后将结果上也显示ADLUBLB的定量任务中只标值上的结果。