Offline benchmarks for microservice root cause analysis (RCA) rank methods by a score pooled over several applications, and a pooled lead leaves two things unstated: on which of the systems it holds, and how the score was produced. We audit both on three public benchmark families (OpenRCA, RCAEval, and PetShop), running the released implementations of 16 methods under recorded configurations on 11 subsystems and on a second release of three of the applications. Whether a pooled lead between published methods holds on every subsystem is decided by a quantity the leaderboard withholds, the spread of the lead across subsystems: of 15 leads whose margin exceeds this spread, 14 hold everywhere; of the 8 whose margin is below it, 6 reverse. The per-fault and per-scenario tables that the benchmark papers publish split the same way, and every error of a ranking computed without one subsystem falls on a lead of the second kind. On the second release a third of the comparisons change sign, most through one method's response to the metric kinds it records, and on several subsystems a simple probe or a shipped baseline beats every published method. How the score was produced matters as much: the input table that one released runner reads is not the one its documentation describes, and this choice alone reverses a comparison between published methods with intervals excluding zero both ways; an unpinned dependency and a duplicated column header each make a method return its input column order without an error. Of 23 RCA papers we examined, none releases the per-case outputs needed to check either premise. We recommend that a pooled comparison state the subsystems it covers and its spread across them, the input table, the handling of failed outputs, and the denominator, and be accompanied by per-case outputs, as ours are in our replication package.

翻译：暂无翻译