In this paper, we present our finding that prepending a Task-Agnostic Prefix Prompt (TAPP) to the input improves the instruction-following ability of various Large Language Models (LLMs) during inference. TAPP is different from canonical prompts for LLMs in that it is a fixed prompt prepended to the beginning of every input regardless of the target task for zero-shot generalization. We observe that both base LLMs (i.e. not fine-tuned to follow instructions) and instruction-tuned models benefit from TAPP, resulting in 34.58% and 12.26% improvement on average, respectively. This implies that the instruction-following ability of LLMs can be improved during inference time with a fixed prompt constructed with simple heuristics. We hypothesize that TAPP assists language models to better estimate the output distribution by focusing more on the instruction of the target task during inference. In other words, such ability does not seem to be sufficiently activated in not only base LLMs but also many instruction-fine-tuned LLMs. All experiments are reproducible from https://github.com/seonghyeonye/TAPP.

翻译：本文发现，在输入前添加一个任务无关的前缀提示（Task-Agnostic Prefix Prompt，TAPP），能够提升多种大型语言模型（LLMs）在推理过程中的指令跟随能力。与常规提示不同，TAPP是一个固定提示，无论目标任务是什么，都会被附加在每个输入的开头，以实现零样本泛化。我们观察到，无论是基础LLM（即未经过指令微调的模型）还是经过指令微调的模型，都能从TAPP中受益，平均分别提升34.58%和12.26%。这表明，通过在推理时使用基于简单启发式方法构建的固定提示，可以增强LLM的指令跟随能力。我们假设，TAPP通过帮助语言模型在推理时更关注目标任务的指令，从而更好地估计输出分布。换言之，这种能力不仅在基础LLM中未被充分激活，在许多指令微调后的LLM中也是如此。所有实验均可通过https://github.com/seonghyeonye/TAPP 复现。