Label scarcity is a bottleneck for improving task performance in specialised domains. We propose a novel compositional transfer learning framework (DoT5 - domain compositional zero-shot T5) for zero-shot domain transfer. Without access to in-domain labels, DoT5 jointly learns domain knowledge (from MLM of unlabelled in-domain free text) and task knowledge (from task training on more readily available general-domain data) in a multi-task manner. To improve the transferability of task training, we design a strategy named NLGU: we simultaneously train NLG for in-domain label-to-data generation which enables data augmentation for self-finetuning and NLU for label prediction. We evaluate DoT5 on the biomedical domain and the resource-lean subdomain of radiology, focusing on NLI, text summarisation and embedding learning. DoT5 demonstrates the effectiveness of compositional transfer learning through multi-task learning. In particular, DoT5 outperforms the current SOTA in zero-shot transfer by over 7 absolute points in accuracy on RadNLI. We validate DoT5 with ablations and a case study demonstrating its ability to solve challenging NLI examples requiring in-domain expertise.

翻译：标签稀缺是提升专业领域任务性能的瓶颈。我们提出了一种新颖的组合式迁移学习框架（DoT5——领域组合式零样本T5），用于零样本领域迁移。在无法获取领域内标签的情况下，DoT5以多任务方式联合学习领域知识（通过无标签领域内自由文本的MLM）和任务知识（通过更易获取的通用领域数据上的任务训练）。为了提升任务训练的可迁移性，我们设计了一种名为NLGU的策略：同时训练NLG进行领域内标签到数据的生成（支持自我微调的数据增强）和NLU进行标签预测。我们在生物医学领域和资源匮乏的放射学子领域上评估了DoT5，重点关注NLI、文本摘要和嵌入学习。DoT5通过多任务学习展示了组合式迁移学习的有效性。特别地，DoT5在RadNLI上的准确率超过当前零样本迁移SOTA达7个绝对百分点。我们通过消融实验和案例研究验证了DoT5解决需要领域专业知识的具有挑战性的NLI示例的能力。