Feature-based anomaly detection is widely adopted in industrial inspection due to the strong representational power of large pre-trained vision encoders. While most existing methods focus on improving within-category anomaly scoring, practical deployments increasingly require task-agnostic operation under continual category expansion, where the category identity is unknown at test time. In this setting, overall performance is often dominated by expert selection, namely routing an input to an appropriate normality model before any head-specific scoring is applied. However, routing rules that compare head-specific anomaly scores across independently constructed heads are unreliable in practice, as score distributions can differ substantially across categories in scale and tail behavior. We propose GCR, a lightweight mixture-of-experts framework for stabilizing task-agnostic continual anomaly detection through geometry-consistent routing. GCR routes each test image directly in a shared frozen patch-embedding space by minimizing an accumulated nearest-prototype distance to category-specific prototype banks, and then computes anomaly maps only within the routed expert using a standard prototype-based scoring rule. By separating cross-head decision making from within-head anomaly scoring, GCR avoids cross-head score comparability issues without requiring end-to-end representation learning. Experiments on MVTec AD and VisA show that geometry-consistent routing substantially improves routing stability and mitigates continual performance collapse, achieving near-zero forgetting while maintaining competitive detection and localization performance. These results indicate that many failures previously attributed to representation forgetting can instead be explained by decision-rule instability in cross-head routing. Code is available at https://github.com/jw-chae/GCR

翻译：基于特征的异常检测凭借大规模预训练视觉编码器的强大表征能力，在工业检测领域得到广泛应用。现有方法多聚焦于提升类别内的异常评分性能，而实际部署场景日益需要支持持续类别扩展下的任务无关操作，即在测试时类别身份未知。在此设定下，整体性能往往由专家选择机制主导——即在执行任何专家特定评分前，将输入样本路由至合适的正常性模型。然而，基于跨独立构建专家间的评分比较路由规则在实践中并不可靠，因为不同类别间的评分分布在量级和尾部行为上可能存在显著差异。本文提出GCR，一种轻量级专家混合框架，通过几何一致性路由实现任务无关持续异常检测的稳定化。GCR在共享的冻结补丁嵌入空间中，通过最小化与类别特定原型库的累积最近邻距离，直接路由每个测试图像，随后仅在被路由的专家内部采用标准的基于原型的评分规则计算异常图。通过将跨专家决策与专家内异常评分解耦，GCR在无需端到端表征学习的前提下，规避了跨专家评分可比性问题。在MVTec AD和VisA数据集上的实验表明，几何一致性路由显著提升了路由稳定性并缓解了持续性能衰退，在保持竞争力的检测与定位性能的同时实现了接近零遗忘。这些结果表明，先前归因于表征遗忘的诸多失效案例，实际上可解释为跨专家路由中决策规则的不稳定性。代码已开源：https://github.com/jw-chae/GCR