The standard way to study Large Language Models (LLMs) with benchmarks or psychology questionnaires is to provide many different queries from similar minimal contexts (e.g. multiple choice questions). However, due to LLMs' highly context-dependent nature, conclusions from such minimal-context evaluations may be little informative about the model's behavior in deployment (where it will be exposed to many new contexts). We argue that context-dependence (specifically, value stability) should be studied a specific property of LLMs and used as another dimension of LLM comparison (alongside others such as cognitive abilities, knowledge, or model size). We present a case-study on the stability of value expression over different contexts (simulated conversations on different topics) as measured using a standard psychology questionnaire (PVQ) and on behavioral downstream tasks. Reusing methods from psychology, we study Rank-order stability on the population (interpersonal) level, and Ipsative stability on the individual (intrapersonal) level. We consider two settings (with and without instructing LLMs to simulate particular personas), two simulated populations, and three downstream tasks. We observe consistent trends in the stability of models and model families - Mixtral, Mistral, GPT-3.5 and Qwen families are more stable than LLaMa-2 and Phi. The consistency of these trends implies that some models exhibit higher value-stability than others, and that value stability can be estimated with the set of introduced methodological tools. When instructed to simulate particular personas, LLMs exhibit low Rank-Order stability, which further diminishes with conversation length. This highlights the need for future research on LLMs that coherently simulate different personas. This paper provides a foundational step in that direction, and, to our knowledge, it is the first study of value stability in LLMs.

翻译：通过基准测试或心理学问卷研究大语言模型（LLMs）的标准方法是提供来自相似最小化语境（例如多项选择题）的许多不同查询。然而，由于LLMs高度依赖情境的特性，基于这类最小化语境评估得出的结论，可能无法有效说明模型在部署场景中的行为表现（那时它将面临众多全新语境）。我们认为，情境依赖性（具体而言是价值稳定性）应被视为LLMs的特定属性，并作为LLM比较的另一个维度（与其他维度如认知能力、知识储备或模型规模并列）。我们通过标准心理学问卷（PVQ）和行为下游任务，对不同情境（不同主题的模拟对话）中价值表达的稳定性进行了案例研究。借鉴心理学方法，我们在群体（人际）层面研究排序稳定性，在个体（个人）层面研究自比稳定性。我们考虑了两种设置（有/无指示LLM模拟特定角色）、两个模拟人群和三项下游任务。我们在模型及模型家族的稳定性上观察到了一致趋势——Mixtral、Mistral、GPT-3.5和Qwen家族比LLaMa-2和Phi更稳定。这些趋势的一致性表明，某些模型表现出比其他模型更高的价值稳定性，并且可以通过引入的方法工具集来评估价值稳定性。当LLMs被指示模拟特定角色时，其排序稳定性较低，并随着对话长度增加进一步降低。这凸显了未来研究需要开发能连贯模拟不同角色的LLMs。本文为该方向提供了基础性探索，据我们所知，这是首次对LLMs中价值稳定性的研究。