Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.

翻译：基准测试通过提供标准化和明确的性能度量，是评估和推进大语言模型（LLM）及多模态大语言模型（MLLM）的基础。然而，基准的构建通常劳动密集且难以复用，引发了可持续性和可扩展性问题。此外，现有基准在发布后往往迅速达到性能饱和，导致对最先进模型的区分能力不足。为解决这些挑战，我们提出了基准智能体（Benchmark Agent），一种全自主的智能体系统，专为基准构建而设计。我们的框架统筹了完整的基准构建流程，从用户查询分析、子任务设计，到数据标注和质量控制。为评估基准智能体，我们将其应用于生成15个代表性基准，涵盖多种评估场景，包括文本理解、多模态理解和特定领域推理。大量实验（包括人工评估、基于大语言模型的裁判评估和一致性检验）表明，基准智能体能在极少人工干预下生成高质量的基准样本。更重要的是，通过持续评估，我们观察到若干深刻发现，例如当前模型在某些特定领域推理任务上存在困难。我们相信快速演进的基准能够为研究社区做出重要贡献。预览和代码将公开在演示页面和代码仓库中。