Large pretrained language models (LLMs) can be rapidly adapted to a wide variety of tasks via a text-to-text approach, where the instruction and input are fed to the model in natural language. Combined with in-context learning (ICL), this paradigm is impressively flexible and powerful. However, it also burdens users with an overwhelming number of choices, many of them arbitrary. Inspired by markup languages like HTML, we contribute a method of using soft-token tags to compose prompt templates. This approach reduces arbitrary decisions and streamlines the application of ICL. Our method is a form of meta-learning for ICL; it learns these tags in advance during a parameter-efficient fine-tuning ``warm-up'' process. The tags can subsequently be used in templates for ICL on new, unseen tasks without any additional fine-tuning. Our experiments with this approach yield promising initial results, improving LLM performance on important enterprise applications such as few-shot and open-world intent detection, as well as text classification in news and legal domains.

翻译：大型预训练语言模型可以通过文本到文本的方式快速适应多种任务，即通过自然语言向模型提供指令和输入。结合上下文学习，这种范式展现出惊人的灵活性和强大能力。然而，这也给用户带来了大量选择负担，其中许多选择具有随意性。受HTML等标记语言的启发，我们提出了一种使用软标记标签组合提示模板的方法。该方法减少了随意性决策，简化了上下文学习的应用流程。我们的方法是一种针对上下文学习的元学习形式：在参数高效微调"预热"阶段提前学习这些标签。这些标签随后可在新任务场景中直接用于构建上下文学习模板，无需额外微调。实验表明，该方法在少样本与开放世界意图检测等关键企业应用场景中取得了初步成效，同时显著提升了新闻和法律领域的文本分类性能。