Large Language Models (LLMs) have achieved remarkable success across various industries due to their exceptional generative capabilities. However, for safe and effective real-world deployments, ensuring honesty and helpfulness is critical. This paper addresses the question: Can we prioritize the helpfulness of LLMs while preserving their honesty? To begin with, we establish exhaustive principles aimed at guaranteeing the honesty of LLM. Additionally, we introduce a novel dataset, referred to as HoneSet, comprising 930 queries spanning six categories meticulously crafted to assess an LLM's capacity for maintaining honesty. Subsequently, we present two approaches to augmenting honesty and helpfulness in LLMs: a training-free enhancement and a fine-tuning-based improvement. The training-free approach, which is based on curiosity-driven prompting, empowers LLMs to articulate internal confusion and uncertainty regarding queries, thereby optimizing their responses. Conversely, the fine-tuning-based method employs a two-stage process inspired by curriculum learning: initially instructing LLMs to discern between honest and dishonest responses, then refining their training to enhance helpfulness. Experiments conducted on nine prominent LLMs demonstrate a significant improvement in alignment with honesty across all models through the implementation of our proposed enhancements. Particularly noteworthy is the 65.3% enhancement observed in Llama3-8b and the remarkable 124.7% improvement in Mistral-7b, as measured by the H$^{2}$ (honest and helpful) assessment. We believe that our work can pave the way for developing more trustworthy LLMs for real-world applications.

翻译：大型语言模型（LLM）凭借其卓越的生成能力，在各行业取得了显著成功。然而，为确保其在现实世界中安全有效地部署，保证其诚实性与有益性至关重要。本文探讨了以下问题：我们能否在保持LLM诚实性的同时，优先提升其有益性？首先，我们建立了一套详尽的准则，旨在保障LLM的诚实性。此外，我们引入了一个新颖的数据集，称为HoneSet，包含涵盖六个类别的930个查询，这些查询经过精心设计，用于评估LLM保持诚实性的能力。随后，我们提出了两种增强LLM诚实性与有益性的方法：一种无需训练的增强方法，以及一种基于微调的改进方法。无需训练的方法基于好奇心驱动的提示技术，使LLM能够针对查询表达内部的困惑与不确定性，从而优化其回应。相反，基于微调的方法采用受课程学习启发的两阶段流程：首先指导LLM区分诚实与不诚实的回应，随后通过训练精炼以提升其有益性。在九个主流LLM上进行的实验表明，通过实施我们提出的增强方案，所有模型在诚实性对齐方面均取得了显著提升。尤其值得注意的是，根据H$^{2}$（诚实与有益）评估，Llama3-8b的评分提升了65.3%，而Mistral-7b更是实现了124.7%的显著改进。我们相信，这项工作能为开发更值得信赖、适用于现实应用的LLM铺平道路。