In-context learning (ICL) enables Large Language Models (LLMs) to perform tasks using few demonstrations, facilitating task adaptation when labeled examples are hard to obtain. However, ICL is sensitive to the choice of demonstrations, and it remains unclear which demonstration attributes enable in-context generalization. In this work, we conduct a perturbation study of in-context demonstrations for low-resource Named Entity Detection (NED). Our surprising finding is that in-context demonstrations with partially correct annotated entity mentions can be as effective for task transfer as fully correct demonstrations. Based off our findings, we propose Pseudo-annotated In-Context Learning (PICLe), a framework for in-context learning with noisy, pseudo-annotated demonstrations. PICLe leverages LLMs to annotate many demonstrations in a zero-shot first pass. We then cluster these synthetic demonstrations, sample specific sets of in-context demonstrations from each cluster, and predict entity mentions using each set independently. Finally, we use self-verification to select the final set of entity mentions. We evaluate PICLe on five biomedical NED datasets and show that, with zero human annotation, PICLe outperforms ICL in low-resource settings where limited gold examples can be used as in-context demonstrations.

翻译：上下文学习（ICL）使大型语言模型（LLM）能够通过少量示例演示执行任务，从而在难以获取标注样本时促进任务适应。然而，ICL对演示示例的选择非常敏感，且目前尚不清楚哪些演示属性能够实现上下文泛化。本研究针对低资源命名实体检测（NED）任务进行了上下文演示的扰动分析。我们意外地发现，仅包含部分正确标注实体提及的上下文演示，在任务迁移效果上可与完全正确的演示相媲美。基于此发现，我们提出了伪标注上下文学习框架PICLe，该框架利用带有噪声的伪标注演示进行上下文学习。PICLe首先通过零样本方式调用LLM对大量演示进行自动标注，随后对这些合成演示进行聚类，并从各聚类中分别采样特定演示集进行独立预测，最后通过自验证机制筛选最终的实体提及集合。我们在五个生物医学NED数据集上评估PICLe，结果表明：在仅能使用有限标准示例作为上下文演示的低资源场景中，PICLe在完全无需人工标注的情况下，其性能显著优于传统ICL方法。