Natural Language Inference (NLI) remains an important benchmark task for LLMs. NLI datasets are a springboard for transfer learning to other semantic tasks, and NLI models are standard tools for identifying the faithfulness of model-generated text. There are several large scale NLI datasets today, and models have improved greatly by hill-climbing on these collections. Yet their realistic performance on out-of-distribution/domain data is less well-understood. We explore the opportunity for synthetic high-quality datasets to adapt NLI models for zero-shot use in downstream applications across new and unseen text domains. We demonstrate a new approach for generating NLI data in diverse domains and lengths, so far not covered by existing training sets. The resulting examples have meaningful premises, the hypotheses are formed in creative ways rather than simple edits to a few premise tokens, and the labels have high accuracy. We show that models trained on this data ($685$K synthetic examples) have the best generalization to completely new downstream test settings. On the TRUE benchmark, a T5-small model trained with our data improves around $7\%$ on average compared to training on the best alternative dataset. The improvements are more pronounced for smaller models, while still meaningful on a T5 XXL model. We also demonstrate gains on test sets when in-domain training data is augmented with our domain-general synthetic data.

翻译：自然语言推理（NLI）仍然是评估大型语言模型（LLM）性能的重要基准任务。NLI数据集是迁移学习至其他语义任务的跳板，而NLI模型已成为识别模型生成文本忠实度的标准工具。当前存在多个大规模NLI数据集，模型通过在这些数据集上进行"爬山式"优化已取得显著进步。然而，模型在分布外/跨领域数据上的实际性能尚未得到充分理解。本研究探索利用高质量合成数据集，使NLI模型能够适应下游应用中全新未见文本领域的零样本使用场景。我们提出了一种在现有训练集尚未覆盖的多样化领域和文本长度中生成NLI数据的新方法。所生成的样本具有语义完整的前提，假设通过创造性方式构建而非对前提词汇的简单编辑，且标注准确率极高。实验表明，基于该数据（$685$K个合成样本）训练的模型在全新下游测试场景中展现出最优泛化能力。在TRUE基准测试中，采用本数据训练的T5-small模型相较于最佳替代数据集训练平均提升约$7\%$。该改进在小型模型中更为显著，而在T5 XXL模型上仍具实际意义。我们还验证了将领域内训练数据与本领域通用合成数据结合时，模型在测试集上的性能提升。