We introduce a linguistically enhanced combination of pre-training methods for transformers. The pre-training objectives include POS-tagging, synset prediction based on semantic knowledge graphs, and parent prediction based on dependency parse trees. Our approach achieves competitive results on the Natural Language Inference task, compared to the state of the art. Specifically for smaller models, the method results in a significant performance boost, emphasizing the fact that intelligent pre-training can make up for fewer parameters and help building more efficient models. Combining POS-tagging and synset prediction yields the overall best results.

翻译：我们引入了一种语言强化的变压器培训前方法组合,培训前目标包括:按静脉拖动、基于语义知识图的合成预测和基于依赖性剖析树的母体预测。我们的方法在自然语言推导任务上取得了与先进水平相比的竞争性结果。具体地说,对于较小的模型来说,该方法可以带来显著的性能提升,强调智能化的预培训可以弥补较少的参数,并有助于建立更有效的模型。将POS拖动和同步预测结合起来可以产生总体的最佳结果。