Zero-shot anomaly localisation via vision-language models (VLMs) offers a compelling approach for rare pathology detection, yet its performance is fundamentally limited by the absence of healthy anatomical context. We reformulate zero-shot localisation as a comparative inference problem in which anomalies are identified through structured comparison against reference distributions of normal anatomy. We introduce WALDO, a training-free framework grounded in optimal transport theory that enables comparative reasoning through: (i) entropy-weighted Sliced Wasserstein distances for anatomically-aware reference selection from DINOv2 patch distributions, (ii) Goldilocks zone sampling exploiting the non-monotonic relationship between reference similarity and localisation accuracy, and (iii) self-consistency aggregation via weighted non-maximum suppression. We theoretically analyse the Goldilocks effect through distributional divergence, and show that references with moderate similarity minimize a bias-variance trade-off in comparative visual reasoning. On the NOVA brain MRI benchmark, WALDO with Qwen2.5-VL-72B achieves $43.5_{\pm1.6}\%$ mAP@30 (95\% CI: [40.4, 46.7]), representing a 19\% relative improvement over zero-shot baselines. Cross-model evaluation shows consistent gains: GPT-4o achieves $32.0_{\pm6.5}\%$ and Qwen3-VL-32B achieves $32.0_{\pm6.6}\%$ mAP@30. Paired McNemar tests confirm statistical significance ($p<0.01$). Source code is available at https://github.com/bkainz/WALDO_MICCAI26_demo .

翻译：通过视觉语言模型进行零样本异常定位为罕见病理检测提供了一种有前景的方法，但其性能根本上受限于缺乏健康解剖背景。我们将零样本定位重新表述为比较推理问题，在该问题中，异常通过与正常解剖参考分布的结构化比较来识别。我们提出WALDO，一个基于最优传输理论无需训练的框架，通过以下方式实现比较推理：(i) 熵加权切片Wasserstein距离，用于从DINOv2块分布中进行解剖感知参考选择；(ii) 利用参考相似性与定位精度之间的非单调关系，采用“金发姑娘”区域采样；以及(iii) 通过加权非极大值抑制进行自一致性聚合。我们通过分布散度理论分析了“金发姑娘”效应，并表明中等相似性的参考可最小化比较视觉推理中的偏差-方差权衡。在NOVA脑部MRI基准上，WALDO搭配Qwen2.5-VL-72B实现了$43.5_{\pm1.6}\%$的mAP@30（95%置信区间：[40.4, 46.7]），较零样本基线相对提升19%。跨模型评估显示一致增益：GPT-4o达到$32.0_{\pm6.5}\%$，Qwen3-VL-32B达到$32.0_{\pm6.6}\%$的mAP@30。配对McNemar检验确认统计显著性（$p<0.01$）。源代码见https://github.com/bkainz/WALDO_MICCAI26_demo。