The ONNX Optimizer, part of the official ONNX repository and widely adopted for graph-level model optimizations, is used by default to optimize ONNX models. Despite its popularity, its ability to preserve model correctness has not been systematically evaluated. We present DiTOX, an automated framework for comprehensively assessing the correctness of the ONNX Optimizer using differential testing, fault localization, and evaluation techniques that generalize to other compiler optimizers. DiTOX applies optimization passes to a corpus of ONNX models, executes both original and optimized versions on user-defined inputs, and detects discrepancies in behavior or optimizer failures. When divergences are observed, DiTOX isolates the responsible optimization pass through iterative, fine-grained analysis. We evaluated DiTOX on 130 models from the ONNX Model Hub spanning vision and language tasks. We found that 9.2% of model instances crashed the optimizer or produced invalid models under default settings. Moreover, output discrepancies occurred in 30% of classification models and 16.6% of object detection and segmentation models, while text-based models were largely robust. Overall, DiTOX uncovered 15 issues -- 14 previously unknown -- affecting 9 of the 47 optimization passes as well as the optimizer infrastructure. All issues were reported to the ONNX Optimizer developers. Our results demonstrate that DiTOX provides a simple and effective approach for validating AI model optimizers and is readily extensible beyond ONNX.

翻译：ONNX优化器作为官方ONNX代码库的组成部分，被广泛用于图级模型优化，并默认用于优化ONNX模型。尽管其应用广泛，但其保持模型正确性的能力尚未得到系统评估。本文提出DiTOX，这是一个通过差分测试、故障定位及可泛化至其他编译器优化器的评估技术，全面评估ONNX优化器正确性的自动化框架。DiTOX对ONNX模型语料库应用优化过程，在用户定义输入上执行原始版本与优化版本，并检测行为差异或优化器故障。当观察到分歧时，DiTOX通过迭代细粒度分析定位责任优化过程。我们在涵盖视觉与语言任务的ONNX模型中心的130个模型上评估DiTOX，发现在默认设置下9.2%的模型实例会导致优化器崩溃或产生无效模型。此外，30%的分类模型以及16.6%的目标检测与分割模型出现输出差异，而基于文本的模型则基本保持稳健。总体而言，DiTOX揭示了15个问题（其中14个为先前未知问题），影响47个优化过程中的9个以及优化器基础设施。所有问题均已向ONNX优化器开发团队报告。我们的结果表明，DiTOX为验证AI模型优化器提供了一种简单有效的方法，并易于扩展至ONNX之外的场景。