In-context learning, which offers substantial advantages over fine-tuning, is predominantly observed in decoder-only models, while encoder-decoder (i.e., seq2seq) models excel in methods that rely on weight updates. Recently, a few studies have demonstrated the feasibility of few-shot learning with seq2seq models; however, this has been limited to tasks that align well with the seq2seq architecture, such as summarization and translation. Inspired by these initial studies, we provide a first-ever extensive experiment comparing the in-context few-shot learning capabilities of decoder-only and encoder-decoder models on a broad range of tasks. Furthermore, we propose two methods to more effectively elicit in-context learning ability in seq2seq models: objective-aligned prompting and a fusion-based approach. Remarkably, our approach outperforms a decoder-only model that is six times larger and exhibits significant performance improvements compared to conventional seq2seq models across a variety of settings. We posit that, with the right configuration and prompt design, seq2seq models can be highly effective few-shot learners for a wide spectrum of applications.

翻译：上下文学习相比于微调具有显著优势，目前主要见于解码器专用模型，而编码器-解码器（即序列到序列）模型在依赖权重更新的方法中表现出众。近期有少量研究证明了序列到序列模型具备小样本学习的可行性，但仅限于与序列到序列架构高度匹配的任务（如摘要生成和翻译）。受这些初步研究启发，我们首次在广泛任务类型上开展大规模实验，系统比较了解码器专用模型与编码器-解码器模型在上下文小样本学习能力上的差异。此外，我们提出两种方法以更有效地激发序列到序列模型的上下文学习能力：目标对齐提示法和融合式方法。值得关注的是，我们的方法性能优于参数量大六倍的解码器专用模型，并在多种设置下相较于传统序列到序列模型展现出显著性能提升。我们认为，通过合理配置与提示设计，序列到序列模型能够成为覆盖广泛应用的强大小样本学习器。