Aligning AI with human values is a pressing unsolved problem. To address the lack of quantitative metrics for value alignment, we propose EigenBench: a black-box method for comparatively benchmarking language models' values. Given an ensemble of models, a constitution describing a value system, and a dataset of scenarios, our method returns a vector of scores quantifying each model's alignment to the given constitution. To produce these scores, each model judges the outputs of other models across many scenarios, and these judgments are aggregated with EigenTrust (Kamvar et al., 2003), yielding scores that reflect a weighted consensus judgment of the whole ensemble. EigenBench uses no ground truth labels, as it is designed to quantify subjective traits for which reasonable judges may disagree on the correct label. Hence, to validate our method, we collect human judgments on the same ensemble of models and show that EigenBench's judgments align closely with those of human evaluators. We further demonstrate that EigenBench can recover model rankings on the GPQA benchmark without access to objective labels, supporting its viability as a framework for evaluating subjective values for which no ground truths exist. The code is available at https://github.com/jchang153/EigenBench.

翻译：使人工智能与人类价值观对齐是一个紧迫且尚未解决的难题。针对价值对齐缺乏量化指标的问题，我们提出了EigenBench：一种用于比较性评估语言模型价值观的黑盒方法。给定一个模型集合、描述价值体系的章程以及场景数据集，我们的方法会返回一个分数向量，用于量化每个模型与给定章程的对齐程度。为生成这些分数，每个模型需在众多场景中评判其他模型的输出，这些评判结果通过EigenTrust（Kamvar等人，2003）进行聚合，从而产生反映整个集合加权共识评判的分数。EigenBench不使用真实标签，因为它旨在量化主观特质——对于这些特质，合理的评判者可能对正确标签存在分歧。因此，为验证我们的方法，我们收集了针对同一模型集合的人类评判，并证明EigenBench的评判结果与人类评估者的评判高度一致。我们进一步证明，EigenBench能够在无需访问客观标签的情况下，恢复模型在GPQA基准测试中的排名，这支持了其作为评估不存在真实标签的主观价值观框架的可行性。代码可在https://github.com/jchang153/EigenBench获取。