We develop evaluation methods for measuring the economic decision-making capabilities and tendencies of LLMs. First, we develop benchmarks derived from key problems in economics -- procurement, scheduling, and pricing -- that test an LLM's ability to learn from the environment in context. Second, we develop the framework of litmus tests, evaluations that quantify an LLM's choice behavior on a stylized decision-making task with multiple conflicting objectives. Each litmus test outputs a litmus score, which quantifies an LLM's tradeoff response, a reliability score, which measures the coherence of an LLM's choice behavior, and a competency score, which measures an LLM's capability at the same task when the conflicting objectives are replaced by a single, well-specified objective. Evaluating a broad array of frontier LLMs, we (1) investigate changes in LLM capabilities and tendencies over time, (2) derive economically meaningful insights from the LLMs' choice behavior and chain-of-thought, (3) validate our litmus test framework by testing self-consistency, robustness, and generalizability. Overall, this work provides a foundation for evaluating LLM agents as they are further integrated into economic decision-making.

翻译：我们开发了用于衡量大型语言模型经济决策能力与倾向的评估方法。首先，我们构建了源自经济学关键问题——采购、调度与定价——的基准测试，以检验LLM在情境中从环境学习的能力。其次，我们提出了石蕊测试框架，该框架通过具有多重冲突目标的程式化决策任务来量化LLM的选择行为。每个石蕊测试输出三个指标：石蕊分数（量化LLM的权衡响应）、可靠性分数（衡量LLM选择行为的一致性）以及能力分数（衡量当冲突目标被单一明确目标取代时，LLM在同一任务中的表现能力）。通过对一系列前沿LLM进行评估，我们（1）探究了LLM能力与倾向随时间的变化，（2）从LLM的选择行为与思维链中提取具有经济学意义的洞见，（3）通过测试自洽性、鲁棒性与泛化性验证了石蕊测试框架。总体而言，本研究为评估日益融入经济决策过程的LLM智能体奠定了方法论基础。