The emergence of generative Large Language Models (LLMs) emphasizes the need for accurate and efficient prompting approaches. LLMs are often applied in Few-Shot Learning (FSL) contexts, where tasks are executed with minimal training data. FSL has become popular in many Artificial Intelligence (AI) subdomains, including AI for health. Rare diseases affect a small fraction of the population. Rare disease identification from clinical notes inherently requires FSL techniques due to limited data availability. Manual data collection and annotation is both expensive and time-consuming. In this paper, we propose Models-Vote Prompting (MVP), a flexible prompting approach for improving the performance of LLM queries in FSL settings. MVP works by prompting numerous LLMs to perform the same tasks and then conducting a majority vote on the resulting outputs. This method achieves improved results to any one model in the ensemble on one-shot rare disease identification and classification tasks. We also release a novel rare disease dataset for FSL, available to those who signed the MIMIC-IV Data Use Agreement (DUA). Furthermore, in using MVP, each model is prompted multiple times, substantially increasing the time needed for manual annotation, and to address this, we assess the feasibility of using JSON for automating generative LLM evaluation.

翻译：生成式大型语言模型（LLMs）的涌现凸显了准确高效提示方法的重要性。LLMs常被应用于少样本学习（FSL）场景，即在极少量训练数据下执行任务。FSL已在多个AI子领域（包括医疗AI）中广泛流行。罕见疾病仅影响少数人群，其临床文本中的识别因数据稀缺而天然需要FSL技术。人工数据收集与标注既昂贵又耗时。本文提出模型投票提示（MVP）方法——一种灵活提示策略，用于提升LLM在FSL场景中的查询性能。MVP通过提示多个LLM执行相同任务，并对输出结果进行多数投票。该方法在单样本罕见疾病识别与分类任务上的表现优于集成中任一单独模型。我们还公开了一个新型少样本学习罕见疾病数据集（需签署MIMIC-IV数据使用协议）。此外，使用MVP时需对每个模型进行多次提示，这将显著增加人工标注时间。为应对此问题，我们评估了利用JSON实现生成式LLM自动评估的可行性。