Numerous benchmarks have been established to assess the performance of foundation models on open-ended question answering, which serves as a comprehensive test of a model's ability to understand and generate language in a manner similar to humans. Most of these works focus on proposing new datasets, however, we see two main issues within previous benchmarking pipelines, namely testing leakage and evaluation automation. In this paper, we propose a novel benchmarking framework, Language-Model-as-an-Examiner, where the LM serves as a knowledgeable examiner that formulates questions based on its knowledge and evaluates responses in a reference-free manner. Our framework allows for effortless extensibility as various LMs can be adopted as the examiner, and the questions can be constantly updated given more diverse trigger topics. For a more comprehensive and equitable evaluation, we devise three strategies: (1) We instruct the LM examiner to generate questions across a multitude of domains to probe for a broad acquisition, and raise follow-up questions to engage in a more in-depth assessment. (2) Upon evaluation, the examiner combines both scoring and ranking measurements, providing a reliable result as it aligns closely with human annotations. (3) We additionally propose a decentralized Peer-examination method to address the biases in a single examiner. Our data and benchmarking results are available at: https://lmexam.com.

翻译：众多基准测试已被建立，用于评估基石模型在开放式问答任务上的表现，这全面测试了模型以类人方式理解和生成语言的能力。然而，大多数现有工作聚焦于提出新数据集，我们观察到先前的基准测试流程存在两个主要问题：测试泄露与评估自动化。本文提出一种新颖的基准测试框架——语言模型作为考官（Language-Model-as-an-Examiner），其中语言模型充当知识渊博的考官，基于自身知识生成问题，并以无参考方式评估回答。该框架支持灵活扩展：不同语言模型均可担任考官角色，且通过引入更多样化的触发主题，问题可被持续更新。为实现更全面、公平的评估，我们设计了三种策略：（1）指导语言模型考官跨多个领域生成问题，以探测模型广泛的知识获取能力，并通过追问进行深度评估；（2）在评估时，考官结合评分与排序两种度量，提供与人工标注高度一致的可靠结果；（3）额外提出去中心化的同伴评估方法（Peer-examination），以缓解单一考官存在的偏差。我们的数据与基准测试结果可通过 https://lmexam.com 获取。