Constructing environments for training and evaluating claw-like agents remains a manual, human-intensive process that does not scale. We argue that what is needed is not just a dataset, but an automated pipeline capable of generating diverse, verified environments on demand. To this end, we introduce ClawEnvKit, an autonomous generation pipeline that instantiates this formalism from natural language descriptions. The pipeline comprises three modules: (1) a parser that extracts structured generation parameters from natural language input; (2) a generator that produces the task specification, tool interface, and scoring configuration; and (3) a validator that enforces feasibility, diversity, structural validity, and internal consistency across the generated environments. Using ClawEnvKit, we construct Auto-ClawEval, the first large-scale benchmark for claw-like agents, comprising 1,040 environments across 24 categories. Empirically, Auto-ClawEval matches or exceeds human-curated environments on coherence and clarity at 13,800x lower cost. Evaluated across 4 model families and 8 agent harness frameworks, we find that harness engineering boosts performance by up to 15.7 percentage points over a bare ReAct baseline, completion remains the primary axis of variation with no model saturating the benchmark, and automated generation enables evaluation at a scale previously infeasible. Beyond static benchmarking, ClawEnvKit enables live evaluation: users describe a desired capability in natural language and obtain a verified environment on demand, turning evaluation into a continuous, user-driven process. The same mechanism serves as an on-demand training environment generator, producing task distributions that adapt to an agent's current weaknesses rather than being bounded by existing user logs.

翻译：构建用于训练和评估爪形智能体的环境目前仍是一个依赖人工的手动过程，难以规模化扩展。我们认为，当前所需的不仅是数据集，更是一个能够按需生成多样化且经过验证的自动化流水线。为此，我们提出ClawEnvKit——一个从自然语言描述出发、实现该形式化过程的自主生成流水线。该流水线由三个模块组成：(1) 解析器，从自然语言输入中提取结构化生成参数；(2) 生成器，生成任务规范、工具接口和评分配置；(3) 验证器，确保所生成环境具备可行性、多样性、结构有效性及内部一致性。通过ClawEnvKit，我们构建了首个大规模爪形智能体基准测试集Auto-ClawEval，涵盖24个类别共1,040个环境。实验表明，Auto-ClawEval在连贯性与清晰度上可匹配或超越人工精心策划的环境，而成本仅为后者的13,800分之一。在4个模型家族和8个智能体框架上的评估显示：框架工程可在裸ReAct基线基础上提升最高15.7个百分点的性能；任务完成度仍是核心差异维度，且尚无模型在该基准上达到饱和；自动化生成使得以往不可行的规模化评估成为可能。除静态基准测试外，ClawEnvKit还支持实时评估：用户以自然语言描述所需能力，即可按需获取经过验证的环境，将评估转变为持续的用户驱动过程。这一机制同样可作为按需训练环境生成器，生成适应智能体当前弱点的任务分布，而非受限于既有用户日志。