Large language models (LLMs) have significantly advanced the field of artificial intelligence. Yet, evaluating them comprehensively remains challenging. We argue that this is partly due to the predominant focus on performance metrics in most benchmarks. This paper introduces CogBench, a benchmark that includes ten behavioral metrics derived from seven cognitive psychology experiments. This novel approach offers a toolkit for phenotyping LLMs' behavior. We apply CogBench to 35 LLMs, yielding a rich and diverse dataset. We analyze this data using statistical multilevel modeling techniques, accounting for the nested dependencies among fine-tuned versions of specific LLMs. Our study highlights the crucial role of model size and reinforcement learning from human feedback (RLHF) in improving performance and aligning with human behavior. Interestingly, we find that open-source models are less risk-prone than proprietary models and that fine-tuning on code does not necessarily enhance LLMs' behavior. Finally, we explore the effects of prompt-engineering techniques. We discover that chain-of-thought prompting improves probabilistic reasoning, while take-a-step-back prompting fosters model-based behaviors.

翻译：大语言模型（LLMs）显著推动了人工智能领域的发展，然而对其进行全面评估仍具挑战性。我们认为，这主要源于多数基准测试过度关注性能指标。本文提出CogBench基准测试，包含源自七项认知心理学实验的十种行为指标。这种创新方法为LLMs的表型分析提供了工具包。我们将CogBench应用于35个LLMs，获取了丰富多样的数据集。通过统计多层级建模技术，我们分析了这些数据，并考虑了特定LLMs微调版本间的嵌套依赖关系。研究突显了模型规模与基于人类反馈的强化学习（RLHF）在提升性能及对齐人类行为中的关键作用。有趣的是，我们发现开源模型比专有模型更少冒险，且代码微调并不必然增强LLMs的行为表现。最后，我们探究了提示工程技巧的影响：链式思维提示能提升概率推理能力，而步步为营提示则促进基于模型的行为。