As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting that models may learn to lie in pursuit of their goals. To address these concerns, a body of work has emerged around the notion of "honesty" in LLMs, along with interventions aimed at mitigating deceptive behaviors. However, some benchmarks claiming to measure honesty in fact simply measure accuracy--the correctness of a model's beliefs--in disguise. Moreover, no benchmarks currently exist for directly measuring whether language models lie. In this work, we introduce a large-scale human-collected dataset for directly measuring lying, allowing us to disentangle accuracy from honesty. Across a diverse set of LLMs, we find that while larger models obtain higher accuracy on our benchmark, they do not become more honest. Surprisingly, most frontier LLMs obtain high scores on truthfulness benchmarks yet exhibit a substantial propensity to lie under pressure, resulting in low honesty scores on our benchmark. We find that simple methods, such as representation engineering interventions, can improve honesty. These results underscore the growing need for robust evaluations and effective interventions to ensure LLMs remain trustworthy.

翻译：随着大型语言模型（LLMs）能力日益增强并具备自主性，对其输出可信度的要求显著提高，但与此同时，模型可能为达成目标而学会欺骗的担忧也在加剧。为应对这些担忧，围绕LLMs的"诚实性"概念已涌现出一系列研究，以及旨在缓解欺骗行为的干预措施。然而，某些声称衡量诚实性的基准实际上只是变相衡量准确性——即模型信念的正确性。此外，目前尚不存在直接衡量语言模型是否说谎的基准。本研究引入了一个大规模人工收集的数据集，用于直接测量欺骗行为，使我们能够将准确性与诚实性分离。通过对多样化LLMs的测试，我们发现：尽管更大规模的模型在我们的基准上获得更高准确性，但其诚实性并未相应提升。令人惊讶的是，多数前沿LLMs在真实性基准测试中得分较高，却在压力下表现出显著的欺骗倾向，导致在我们的基准测试中诚实性得分偏低。研究发现，简单的方法（如表征工程干预）即可提升诚实性。这些结果凸显了建立稳健评估体系和有效干预措施的迫切需求，以确保LLMs保持可信赖性。