This paper presents AutoHint, a novel framework for automatic prompt engineering and optimization for Large Language Models (LLM). While LLMs have demonstrated remarkable ability in achieving high-quality annotation in various tasks, the key to applying this ability to specific tasks lies in developing high-quality prompts. Thus we propose a framework to inherit the merits of both in-context learning and zero-shot learning by incorporating enriched instructions derived from input-output demonstrations to optimize original prompt. We refer to the enrichment as the hint and propose a framework to automatically generate the hint from labeled data. More concretely, starting from an initial prompt, our method first instructs a LLM to deduce new hints for selected samples from incorrect predictions, and then summarizes from per-sample hints and adds the results back to the initial prompt to form a new, enriched instruction. The proposed method is evaluated on the BIG-Bench Instruction Induction dataset for both zero-shot and few-short prompts, where experiments demonstrate our method is able to significantly boost accuracy for multiple tasks.

翻译：本文提出AutoHint，一种面向大语言模型（LLM）的自动提示工程与优化新框架。尽管LLM在各类任务中展现出生成高质量标注的卓越能力，但将此能力迁移至特定任务的关键在于开发高质量提示。为此，我们提出一种融合上下文学习与零样本学习优点的框架，通过从输入-输出示例中提取的增强指令来优化原始提示。我们将这种增强称为"提示提示（hint）"，并提出从标注数据自动生成提示提示的框架。具体而言，我们的方法首先以初始提示为基础，指导LLM从错误预测的样本中推导新提示提示，随后对逐样本提示提示进行归纳，并将结果补充到初始提示中以形成增强指令。该方法在BIG-Bench指令归纳数据集上分别针对零样本与小样本提示场景进行评估，实验表明该方法能显著提升多个任务的准确率。