Constructing environments for training and evaluating claw-like agents remains a manual, human-intensive process that does not scale. We argue that what is needed is not just a dataset, but an automated pipeline capable of generating diverse, verified environments on demand. To this end, we introduce ClawEnvKit, an autonomous generation pipeline that instantiates this formalism from natural language descriptions. The pipeline comprises three modules: (1) a parser that extracts structured generation parameters from natural language input; (2) a generator that produces the task specification, tool interface, and scoring configuration; and (3) a validator that enforces feasibility, diversity, structural validity, and internal consistency across the generated environments. Using ClawEnvKit, we construct Auto-ClawEval, the first large-scale benchmark for claw-like agents, comprising 1,040 environments across 24 categories. Empirically, Auto-ClawEval matches or exceeds human-curated environments on coherence and clarity at 13,800x lower cost. Evaluated across 4 model families and 8 agent harness frameworks, we find that harness engineering boosts performance by up to 15.7 percentage points over a bare ReAct baseline, completion remains the primary axis of variation with no model saturating the benchmark, and automated generation enables evaluation at a scale previously infeasible. Beyond static benchmarking, ClawEnvKit enables live evaluation: users describe a desired capability in natural language and obtain a verified environment on demand, turning evaluation into a continuous, user-driven process. The same mechanism serves as an on-demand training environment generator, producing task distributions that adapt to an agent's current weaknesses rather than being bounded by existing user logs.

翻译：为训练与评估爪状智能体构建环境仍是一项依赖人工的手动过程，难以规模化扩展。我们认为关键不在于单一数据集，而在于能够按需生成多样化且经过验证环境的自动化流程。为此，我们提出ClawEnvKit——一种从自然语言描述实例化上述形式化框架的自主生成流程。该流程包含三个模块：（1）解析器，从自然语言输入中提取结构化生成参数；（2）生成器，产生任务规范、工具接口与评分配置；（3）验证器，确保所生成环境的可行性、多样性、结构有效性及内部一致性。利用ClawEnvKit，我们构建了首个面向爪状智能体的大规模基准Auto-ClawEval，包含覆盖24个类别的1040个环境。实验表明，Auto-ClawEval在连贯性与清晰度方面达到甚至超越人工策划的环境，而成本仅为后者的1/13800。在4个模型家族与8个智能体框架上的评估显示：框架工程可使性能相较裸ReAct基线提升最高15.7个百分点；完成度始终是主要变异维度，且无模型在该基准上达到饱和；自动生成使评估规模达到先前不可行的水平。除静态基准测试外，ClawEnvKit还支持实时评估：用户以自然语言描述所需能力后即可按需获取验证过的环境，将评估转化为持续的用户驱动过程。该机制同时可作为按需训练环境生成器，产生适应当前智能体弱点的任务分布——而非受限于现有用户日志。