LLM-based tools are automating more software development tasks at a rapid pace, but there is no rigorous way to evaluate how different architectural choices -- prompts, skills, tools, multi-agent setups -- materially affect both capability and cost. This paper introduces Scylla, an evaluation framework for benchmarking agentic coding tools through structured ablation studies that uses seven testing tiers (T0-T6) progressively adding complexity to isolate what directly influences results and how. The key metric is Cost-of-Pass (CoP): the expected dollar cost to get one correct solution, which directly quantifies the trade-off between complexity and efficiency. The framework is model-agnostic, designed to work with any CLI tool; this paper demonstrates it with Claude Sonnet 4.5, using multiple LLM judges (Opus 4.5, Sonnet 4.5, Haiku 4.5) from the same vendor for evaluation consensus, where judges score results using direct tests, human-designed LLM-evaluated rubrics, and qualitative assessment. The result is a reproducible framework that quantifies trade-offs between agent complexity and actual outcomes, suggesting that architectural complexity does not always improve quality.

翻译：基于大型语言模型（LLM）的工具正在快速自动化更多软件开发任务，但目前缺乏严谨的方法来评估不同架构选择——如提示词、技能、工具、多智能体设置——如何实质性地影响能力与成本。本文介绍斯库拉（Scylla），一个通过结构化消融实验对智能编码工具进行基准测试的评估框架。该框架采用七个测试层级（T0-T6），逐步增加复杂度以隔离直接影响结果的因素及其作用机制。核心指标为通过成本（Cost-of-Pass, CoP）：获得一个正确解决方案的预期美元成本，该指标直接量化了复杂度与效率之间的权衡。该框架与模型无关，可适配任何命令行工具；本文使用Claude Sonnet 4.5进行演示，并采用同一供应商的多个LLM评估器（Opus 4.5、Sonnet 4.5、Haiku 4.5）达成评估共识，评估器通过直接测试、人工设计的LLM评估量规和定性分析对结果进行评分。最终构建出一个可复现的框架，能够量化智能体复杂度与实际结果间的权衡关系，结果表明架构复杂度的提升并不总能改善输出质量。