Large Language Models (LLMs) like gpt-3.5-turbo and claude-instant-1.2 have become instrumental in interpreting and executing semantic-based tasks. Unfortunately, these models' inherent biases, akin to human cognitive biases, adversely affect their performance. Particularly affected is object selection from lists; a fundamental operation in digital navigation and decision-making. This research critically examines these biases and quantifies the effects on a representative list selection task. To explore these biases, we conducted a series of controlled experiments, manipulating temperature, list length, object identity, object type, prompt complexity, and model. This enabled us to isolate and measure the influence of the biases on selection behavior. Our findings show that bias structure is strongly dependent on the model, with object type modulating the magnitude of the effect. With a strong primacy effect, causing the first objects in a list to be disproportionately represented in outputs. Furthermore the usage of guard rails, a prompt engineering method of ensuring a response structure, can increase bias and decrease instruction adherence when combined with a selection task. The bias is ablated when the guard rail step is separated from the list sampling step, lowering the complexity of each individual task. The implications of this research are two-fold, practically providing a guide for designing unbiased LLM applications and theoretically suggesting that LLMs experience a form of cognitive load compensated for by increasing bias.

翻译：大语言模型（如gpt-3.5-turbo和claude-instant-1.2）已成为理解和执行语义任务的关键工具。不幸的是，这些模型固有的偏差（类似于人类的认知偏差）会对其性能产生负面影响。特别受影响的是从列表中选择对象，这是数字导航与决策中的基础操作。本研究严谨审视了这些偏差，并量化了其在代表性列表选择任务中的影响。为了探索这些偏差，我们开展了一系列受控实验，通过操纵温度、列表长度、对象身份、对象类型、提示复杂度以及模型类型，来分离并测量偏差对选择行为的作用。结果表明，偏差结构强烈依赖于模型，且对象类型会调节这种效应的幅度。强烈的首因效应导致列表中的首个对象在输出中被过度呈现。此外，使用“护栏”（一种确保响应结构的提示工程方法）会在与选择任务结合时增加偏差并降低指令遵循度。当将护栏步骤与列表采样步骤分离，从而降低每个单独任务的复杂度时，偏差被消除。本研究具有双重意义：实践上为设计无偏的大语言模型应用提供指南，理论上则暗示大语言模型会经历某种形式的认知负荷，并通过增加偏差来进行补偿。