Humans are influenced by how information is presented, a phenomenon known as the framing effect. Prior work suggests that LLMs may also be susceptible to framing, but it has relied on synthetic data and did not compare to human behavior. To address this gap, we introduce WildFrame - a dataset for evaluating LLM responses to positive and negative framing in naturally-occurring sentences, alongside human responses on the same data. WildFrame consists of 1,000 real-world texts selected to convey a clear sentiment; we then reframe each text in either a positive or negative light and collect human sentiment annotations. Evaluating eleven LLMs on WildFrame, we find that all models respond to reframing in a human-like manner ($r\geq0.52$), and that both humans and models are influenced more by positive than negative reframing. Notably, GPT models are the least correlated with human behavior among all tested models. These findings raise a discussion around the goals of state-of-the-art LLM development and whether models should align closely with human behavior, to preserve cognitive phenomena such as the framing effect, or instead mitigate such biases in favor of fairness and consistency.

翻译：人类会受到信息呈现方式的影响，这种现象被称为框架效应。先前的研究表明，大型语言模型（LLMs）可能也容易受到框架的影响，但这些研究依赖于合成数据，并且没有与人类行为进行比较。为了填补这一空白，我们引入了WildFrame——一个用于评估LLMs对自然发生句子中正面和负面框架的反应的数据集，同时包含人类在同一数据上的反应。WildFrame包含1000个选自现实世界、旨在传达清晰情感的文本；然后，我们将每个文本以正面或负面的方式进行重构，并收集人类的情感标注。在WildFrame上评估十一个LLMs后，我们发现所有模型都以类似人类的方式对重构做出反应（$r\geq0.52$），并且人类和模型都更多地受到正面而非负面重构的影响。值得注意的是，在所有测试模型中，GPT模型与人类行为的相关性最低。这些发现引发了关于最先进LLM开发目标的讨论：模型是否应该与人类行为紧密对齐，以保留如框架效应这样的认知现象，还是应该减轻此类偏见以促进公平性和一致性。