Reliable evaluation is essential for developing and deploying large language models, yet in practice it often requires substantial manual effort: practitioners must identify appropriate benchmarks, reproduce heterogeneous evaluation codebases, configure dataset schema mappings, and interpret aggregated metrics. To address these challenges, we present One-Eval, an agentic evaluation system that converts natural-language evaluation requests into executable, traceable, and customizable evaluation workflows. One-Eval integrates (i) NL2Bench for intent structuring and personalized benchmark planning, (ii) BenchResolve for benchmark resolution, automatic dataset acquisition, and schema normalization to ensure executability, and (iii) Metrics \& Reporting for task-aware metric selection and decision-oriented reporting beyond scalar scores. The system further incorporates human-in-the-loop checkpoints for review, editing, and rollback, while preserving sample evidence trails for debugging and auditability. Experiments show that One-Eval can execute end-to-end evaluations from diverse natural-language requests with minimal user effort, supporting more efficient and reproducible evaluation in industrial settings. Our framework is publicly available at https://github.com/OpenDCAI/One-Eval.

翻译：可靠评估对于大语言模型的开发与部署至关重要，然而实践中常需大量人工投入：从业者需确定合适的基准测试、复现异构评估代码库、配置数据集模式映射并解释聚合指标。为应对这些挑战，我们提出One-Eval——一种将自然语言评估请求转化为可执行、可追溯且可定制评估流程的代理系统。One-Eval集成三大模块：(i) NL2Bench用于意图结构化与个性化基准规划，(ii) BenchResolve实现基准解析、自动数据获取与模式规范化以确保可执行性，(iii) 度量与报告模块支持任务感知的指标选择及超越标量分数的决策导向报告。系统进一步融入人机协同检查点以支持审核、编辑与回滚，同时保留样本证据链用于调试与审计。实验表明，One-Eval能以最小用户投入执行多样化自然语言请求的端到端评估，为工业场景提供更高效且可复现的评估方案。本框架已在https://github.com/OpenDCAI/One-Eval公开。