Large language models (LLM) have recently shown the extraordinary ability to perform unseen tasks based on few-shot examples provided as text, also known as in-context learning (ICL). While recent works have attempted to understand the mechanisms driving ICL, few have explored training strategies that incentivize these models to generalize to multiple tasks. Multi-task learning (MTL) for generalist models is a promising direction that offers transfer learning potential, enabling large parameterized models to be trained from simpler, related tasks. In this work, we investigate the combination of MTL with ICL to build models that efficiently learn tasks while being robust to out-of-distribution examples. We propose several effective curriculum learning strategies that allow ICL models to achieve higher data efficiency and more stable convergence. Our experiments reveal that ICL models can effectively learn difficult tasks by training on progressively harder tasks while mixing in prior tasks, denoted as mixed curriculum in this work. Our code and models are available at https://github.com/harmonbhasin/curriculum_learning_icl .

翻译：大型语言模型（LLM）近期展现出基于文本提供的少量示例执行未见任务的非凡能力，即上下文学习（ICL）。虽然近期研究尝试理解驱动ICL的机制，但鲜有工作探索激励这些模型泛化至多任务的训练策略。面向通用模型的多任务学习（MTL）是一个前景广阔的方向，其具备迁移学习潜力，使大规模参数化模型能够通过更简单的相关任务进行训练。本研究探讨了MTL与ICL的结合，以构建既能高效学习任务又能对分布外样例保持鲁棒性的模型。我们提出了若干有效的课程学习策略，使ICL模型能够实现更高的数据效率和更稳定的收敛。实验表明，通过逐步训练难度递增的任务并混入先前的任务（本文称为混合课程），ICL模型可以有效学习困难任务。我们的代码和模型已开源在https://github.com/harmonbhasin/curriculum_learning_icl。