An Achilles heel of Large Language Models (LLMs) is their tendency to hallucinate non-factual statements. A response mixed of factual and non-factual statements poses a challenge for humans to verify and accurately base their decisions on. To combat this problem, we propose Highlighted Chain-of-Thought Prompting (HoT), a technique for prompting LLMs to generate responses with XML tags that ground facts to those provided in the query. That is, given an input question, LLMs would first re-format the question to add XML tags highlighting key facts, and then, generate a response with highlights over the facts referenced from the input. Interestingly, in few-shot settings, HoT outperforms vanilla chain of thought prompting (CoT) on a wide range of 17 tasks from arithmetic, reading comprehension to logical reasoning. When asking humans to verify LLM responses, highlights help time-limited participants to more accurately and efficiently recognize when LLMs are correct. Yet, surprisingly, when LLMs are wrong, HoTs tend to make users believe that an answer is correct.

翻译：大型语言模型（LLM）的一个致命弱点是其倾向于产生非事实性陈述。混杂事实与非事实陈述的回应对人类验证并据此准确决策构成了挑战。为解决此问题，我们提出高亮思维链提示（HoT），该技术通过提示LLM生成带有XML标签的响应，将事实锚定至查询中提供的信息。具体而言，给定输入问题时，LLM首先会重新格式化问题并添加用于突出关键事实的XML标签，随后生成在输入事实引用处带有高亮标记的回应。值得注意的是，在少样本设定下，HoT在涵盖算术、阅读理解到逻辑推理的17项广泛任务中均优于原始思维链提示（CoT）。当要求人类验证LLM响应时，高亮标记能帮助时间受限的参与者更准确高效地识别LLM的正确输出。然而令人惊讶的是，当LLM出现错误时，HoT反而容易使用户相信错误答案是正确的。