The core assumption behind using mutants as substitutes for real faults is that they represent them faithfully. In traditional software testing, this representativeness has been empirically studied by examining the behavioral relationship between mutant detection and real fault detection. In deep learning (DL), mutants are generated by applying mutation operators to either pre-training or post-training artifacts. However, the realism of mutants in DL remains largely unverified. To the best of our knowledge, this study presents the first empirical comparison of pre-training and post-training mutation approaches in DL with respect to realism. We introduce a statistical framework that quantifies realism using two complementary metrics: coupling strength and behavioral similarity to real faults. We evaluate the framework on four publicly available bug datasets: CleanML, DeepFD, DeepLocalize, and defect4ML. Mutants are generated using state-of-the-art tools representing both mutation approaches. Results show that pre-training mutants exhibit significantly stronger coupling and higher behavioral similarity to real faults than post-training mutants, indicating greater realism. Operator-level analysis shows that this advantage is systematic across individual mutation operators and ranks the operators by realism within each approach. However, the substantial computational cost of pre-training mutation underscores the need for more effective post-training operators that match or exceed the realism of pre-training mutants.

翻译：暂无翻译