We present a controlled experiment evaluating 12 multi-agent LLM collaboration topologies for software architecture design. Using a $2\times2\times2$ factorial design (Authority $\times$ Roles $\times$ Dynamics), we conducted 520 experimental runs across 8 design tasks of varying complexity, with 5 repetitions each. Designs were evaluated on a 12-dimensional rubric by three independent automated evaluators (GPT-OSS 120B, Claude Opus 4.6, Claude Sonnet 4.6). We report four core findings. First, structural adversarial (v4b) ranks #1 by ensemble -- a prompt-engineered adversarial variant that demands rewrite mandates rather than patches (weighted ensemble: 4.637/5.0). Second, cross-model review wins unanimously at #2 -- generate with one model, review with another -- ranking #2 by all three evaluators (weighted ensemble: 4.606). Third, evaluator diversity is itself a finding -- all three evaluators agree v4b is best and v3 is worst, but disagree sharply on v2b (Claude d=1.44 vs. GPT-OSS d=0.45), revealing how different model families weight design qualities. Fourth, parallel merge is fundamentally broken -- all three evaluators place merge variants in the bottom tier (3.65-3.79), due to token starvation and the Frankenstein effect. The weighted ensemble ($2\times$Opus + $2\times$Sonnet + $1\times$GPT-OSS) provides robust rankings across 520 runs, confirmed through independent cross-validation.
翻译:我们提出了一项受控实验,评估了12种用于软件架构设计的多智能体LLM协作拓扑。采用$2\times2\times2$因子设计(权威性$\times$角色$\times$动态性),我们在8个不同复杂度的设计任务上进行了520次实验运行,每个任务重复5次。设计质量由三个独立自动评估器(GPT-OSS 120B、Claude Opus 4.6、Claude Sonnet 4.6)按照12维评分标准进行评价。我们报告四项核心发现。第一,结构化对抗型(v4b)在集成评分中排名第一——这是一种提示工程优化的对抗变体,要求强制重写而非修补(加权集成:4.637/5.0)。第二,跨模型评审以全票优势位列第二——即用某一模型生成、用另一模型评审——在所有三个评估器中均排名第二(加权集成:4.606)。第三,评估器多样性本身即为一项发现——三个评估器一致认为v4b最佳、v3最差,但对v2b评价存在显著分歧(Claude d=1.44 vs. GPT-OSS d=0.45),揭示了不同模型家族对设计质量的权重差异。第四,并行合并存在根本性缺陷——所有评估器均将合并变体置于底层(3.65-3.79),原因在于令牌匮乏和弗兰肯斯坦效应。加权集成评估($2\times$Opus + $2\times$Sonnet + $1\times$GPT-OSS)在520次运行中提供了稳健的排名,并通过独立交叉验证得到确认。