In-context learning (ICL) enables Large Language Models (LLMs) to perform tasks using few demonstrations, facilitating task adaptation when labeled examples are hard to obtain. However, ICL is sensitive to the choice of demonstrations, and it remains unclear which demonstration attributes enable in-context generalization. In this work, we conduct a perturbation study of in-context demonstrations for low-resource Named Entity Detection (NED). Our surprising finding is that in-context demonstrations with partially correct annotated entity mentions can be as effective for task transfer as fully correct demonstrations. Based off our findings, we propose Pseudo-annotated In-Context Learning (PICLe), a framework for in-context learning with noisy, pseudo-annotated demonstrations. PICLe leverages LLMs to annotate many demonstrations in a zero-shot first pass. We then cluster these synthetic demonstrations, sample specific sets of in-context demonstrations from each cluster, and predict entity mentions using each set independently. Finally, we use self-verification to select the final set of entity mentions. We evaluate PICLe on five biomedical NED datasets and show that, with zero human annotation, PICLe outperforms ICL in low-resource settings where limited gold examples can be used as in-context demonstrations.

翻译：上下文学习（ICL）使大型语言模型（LLM）能够通过少量示例演示执行任务，从而在难以获取标注样本时促进任务适应。然而，ICL对演示示例的选择较为敏感，且目前尚不清楚哪些演示属性能够实现上下文泛化。本研究针对低资源命名实体检测（NED）任务进行了上下文演示的扰动分析。我们意外地发现，仅包含部分正确标注实体指称的上下文演示在任务迁移效果上可与完全正确的演示相媲美。基于此发现，我们提出了伪标注上下文学习框架PICLe，该框架利用带噪声的伪标注演示进行上下文学习。PICLe首先通过零样本初筛利用LLM自动标注大量演示样本，随后对这些合成演示进行聚类，从每个聚类中采样特定集合作为上下文演示，并基于各集合独立预测实体指称。最后通过自验证机制筛选最终实体指称集合。我们在五个生物医学NED数据集上评估PICLe，结果表明：在仅能使用有限标准示例作为上下文演示的低资源场景中，PICLe在完全无需人工标注的情况下，其性能优于传统ICL方法。