In day-to-day communication, people often approximate the truth - for example, rounding the time or omitting details - in order to be maximally helpful to the listener. How do large language models (LLMs) handle such nuanced trade-offs? To address this question, we use psychological models and experiments designed to characterize human behavior to analyze LLMs. We test a range of LLMs and explore how optimization for human preferences or inference-time reasoning affects these trade-offs. We find that reinforcement learning from human feedback improves both honesty and helpfulness, while chain-of-thought prompting skews LLMs towards helpfulness over honesty. Finally, GPT-4 Turbo demonstrates human-like response patterns including sensitivity to the conversational framing and listener's decision context. Our findings reveal the conversational values internalized by LLMs and suggest that even these abstract values can, to a degree, be steered by zero-shot prompting.

翻译：在日常沟通中，人们常常会为了最大程度地帮助听者而近似地表达事实——例如，将时间四舍五入或省略细节。大型语言模型如何处理这种微妙的权衡？为解答这一问题，我们采用用于刻画人类行为的心理学模型和实验来分析LLMs。我们测试了一系列LLMs，并探究针对人类偏好的优化或推理时推理如何影响这些权衡。研究发现，基于人类反馈的强化学习同时提升了诚实性与有用性，而思维链提示则使LLMs偏向有用性而非诚实性。最终，GPT-4 Turbo展现出类似人类的响应模式，包括对对话框架和听者决策情境的敏感性。我们的发现揭示了LLMs内化的对话价值观，并表明即使是这些抽象价值观，在某种程度上也可通过零样本提示加以引导。