Blockwise distillation methods for architecture search, such as DNA, DONNA, LANA and Puzzle, score each candidate replacement block in isolation and combine the scores additively. We ask at which budgets this additivity assumption fails, whether fine-tuning repairs the failure, what diagnoses it, and how it can be repaired cheaply. On DeiT-S, DeiT-B, Swin-T and Qwen2.5-0.5B with seven latency budgets each, an additive integer program trails a pairwise interaction-aware program after fine-tuning by at most 1.6 top-1 points on ImageNet-1k at the mildest budgets, but by up to 17.5 points, and a factor of 1.9 in perplexity, at aggressive ones. After 10,000 fine-tuning steps 3.3-9.9 points remain on DeiT, whereas on Swin-T the gap shrinks to at most about 1.8 points within 500 steps. In a learning-rate sweep with the standard 2,000 steps, 6.0-10.7 points remain on DeiT when each method's rate is chosen by its evaluation-set accuracy. An intervention that keeps the block library and only forbids more than two removals in any three adjacent layers shrinks the DeiT gap from 5.8-17.5 to 0.7-5.2 points, mostly by improving the additive solution: adjacent removals are a major failure pattern. Splitting the additivity error into a metric and a functional part, relative to additive logit changes, indicates where a cheap logit-sum predictor can be trusted. A hybrid search that ranks moves with this predictor and measures only the best needs no pair measurements and, after Holm adjustment, is significantly better than the pairwise program in 10 of 14 ImageNet-1k settings, with no statistically significant losses. An ablation shows that its ranking helps except at the most aggressive DeiT-B budget, where the predictor misleads and random ranking or ten-times-costlier evolution does better.
翻译:暂无翻译