Recent research has shown that multi-task pre-training greatly improves the model's robustness and transfer ability, which is crucial for building a high-quality dialog system. However, most previous works on multi-task pre-training rely heavily on human-defined input format or prompt, which is not optimal in quality and quantity. In this work, we propose to use Task-based Automatic Prompt generation (TAP) to automatically generate high-quality prompts. Using the high-quality prompts generated, we scale the corpus of the pre-trained conversation model to 122 datasets from 15 dialog-related tasks, resulting in Universal Pre-trained Conversation Model (UniPCM), a powerful foundation model for various conversational tasks and different dialog systems. Extensive experiments have shown that UniPCM is robust to input prompts and capable of various dialog-related tasks. Moreover, UniPCM has strong transfer ability and excels at low resource scenarios, achieving SOTA results on 9 different datasets ranging from task-oriented dialog to open-domain conversation. Furthermore, we are amazed to find that TAP can generate prompts on par with those collected with crowdsourcing. The code is released with the paper.

翻译：近期研究表明，多任务预训练显著提升了模型的鲁棒性和迁移能力，这对构建高质量对话系统至关重要。然而，现有大多数多任务预训练工作严重依赖人工定义的输入格式或提示，其在质量和数量上均非最优。本文提出基于任务的自动提示生成（Task-based Automatic Prompt generation, TAP），自动生成高质量提示。利用所生成的高质量提示，我们将预训练对话模型的语料库扩展至涵盖15个对话相关任务的122个数据集，由此构建了通用预训练对话模型（Universal Pre-trained Conversation Model, UniPCM）——一个面向各类对话任务与不同对话系统的强大基础模型。大量实验表明，UniPCM对输入提示具有鲁棒性，并能胜任多种对话相关任务。此外，UniPCM具备强大的迁移能力，在低资源场景下表现优异，在涵盖任务导向型对话与开放域对话的9个不同数据集上达到当前最优（SOTA）结果。更令人惊叹的是，我们发现TAP能够生成与通过众包收集的提示质量相当的结果。代码随论文一同发布。