Negation is a core construction in natural language. Despite being very successful on many tasks, state-of-the-art pre-trained language models often handle negation incorrectly. To improve language models in this regard, we propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences from a raw text corpus. By training BERT with the resulting combined objective we reduce the mean top~1 error rate to 4% on the negated LAMA dataset. We also see some improvements on the negated NLI benchmarks.

翻译：否定是自然语言的核心构造。 尽管在很多任务上非常成功, 最先进的预先培训语言模式往往会错误地处理否定问题。 在这方面,为了改进语言模式,我们建议增加语言模型目标, 其目标不能实现, 其依据是原始文本中否定的通用句子。 通过培训BERT 及其共同目标, 我们把否定的LAMA数据集中最高的- 1 误差率降低到4% 。 我们还看到否定的NLI 基准有所改进 。