Large language models (LLMs) have shown impressive capabilities across various tasks, but their performance on domain-specific tasks remains limited. While methods like retrieval augmented generation and fine-tuning can help to address this, they require significant resources. In-context learning (ICL) is a cheap and efficient alternative but cannot match the accuracies of advanced methods. We present Ensemble SuperICL, a novel approach that enhances ICL by leveraging the expertise of multiple fine-tuned small language models (SLMs). Ensemble SuperICL achieves state of the art (SoTA) results on several natural language understanding benchmarks. Additionally, we test it on a medical-domain labelling task and showcase its practicality by using off-the-shelf SLMs fine-tuned on a general language task, achieving superior accuracy in large-scale data labelling compared to all baselines. Finally, we conduct an ablation study and sensitivity analyses to elucidate the underlying mechanism of Ensemble SuperICL. Our research contributes to the growing demand for efficient domain specialisation methods in LLMs, offering a cheap and effective method for practitioners.

翻译：大型语言模型（LLMs）在各种任务中展现出卓越能力，但其在领域特定任务上的表现仍存在局限。尽管检索增强生成和微调等方法有助于解决这一问题，但它们需要大量资源。上下文学习（ICL）是一种经济高效的替代方案，但其准确性无法与先进方法相媲美。本文提出Ensemble SuperICL——一种通过整合多个微调后的小型语言模型（SLMs）的专业知识来增强ICL的新方法。Ensemble SuperICL在多个自然语言理解基准测试中取得了最先进的成果。此外，我们在医学领域标注任务上对其进行了测试，并展示了其实用性：通过使用在通用语言任务上微调的现成SLMs，在大规模数据标注中实现了超越所有基线的准确率。最后，我们通过消融实验和敏感性分析阐明了Ensemble SuperICL的内在机制。本研究响应了LLMs领域对高效领域专业化方法日益增长的需求，为实践者提供了一种经济有效的解决方案。