Natural Language Inference (NLI) remains an important benchmark task for LLMs. NLI datasets are a springboard for transfer learning to other semantic tasks, and NLI models are standard tools for identifying the faithfulness of model-generated text. There are several large scale NLI datasets today, and models have improved greatly by hill-climbing on these collections. Yet their realistic performance on out-of-distribution/domain data is less well-understood. We present an in-depth exploration of the problem of domain generalization of NLI models. We demonstrate a new approach for generating synthetic NLI data in diverse domains and lengths, so far not covered by existing training sets. The resulting examples have meaningful premises, the hypotheses are formed in creative ways rather than simple edits to a few premise tokens, and the labels have high accuracy. We show that models trained on this data ($685$K synthetic examples) have the best generalization to completely new downstream test settings. On the TRUE benchmark, a T5-small model trained with our data improves around $7\%$ on average compared to training on the best alternative dataset. The improvements are more pronounced for smaller models, while still meaningful on a T5 XXL model. We also demonstrate gains on test sets when in-domain training data is augmented with our domain-general synthetic data.

翻译：自然语言推理（NLI）仍是LLM的重要基准任务。NLI数据集是迁移学习到其他语义任务的跳板，NLI模型则是识别模型生成文本忠实度的标准工具。当前存在多个大规模NLI数据集，模型通过在这些集合上的爬山优化已取得显著进步。然而，模型在域外/跨领域数据上的实际表现仍缺乏深入认知。本文对NLI模型的领域泛化问题进行了深度探索，提出了一种在现有训练集尚未覆盖的多领域、多长度范围内的合成NLI数据生成新方法。生成的示例具有语义合理的假设前提，假设形成方式富有创造性（而非对前提中若干词元的简单修改），且标签准确率极高。实验表明，基于该数据（68.5万条合成样本）训练的模型在全新下游测试场景中表现出最优泛化能力。在TRUE基准测试中，使用本数据训练的T5-small模型相比最优替代数据集训练方案平均提升约7%。这种提升对小规模模型更为显著，而在T5 XXL模型上仍具实际意义。此外，当领域内训练数据经过本领域泛化合成数据增强后，模型在测试集上的表现同样获得增益。