A recent Dwarkesh Patel podcast with John Collison and Elon Musk featured an interesting puzzle from Jane Street: they trained a neural net, shuffled all 96 layers, and asked to put them back in order. Given unlabelled layers of a Residual Network and its training dataset, we recover the exact ordering of the layers. The problem decomposes into pairing each block's input and output projections ($48!$ possibilities) and ordering the reassembled blocks ($48!$ possibilities), for a combined search space of $(48!)^2 \approx 10^{122}$, which is more than the atoms in the observable universe. We show that stability conditions during training like dynamic isometry leave the product $W_{\text{out}} W_{\text{in}}$ for correctly paired layers with a negative diagonal structure, allowing us to use diagonal dominance ratio as a signal for pairing. For ordering, we seed-initialize with a rough proxy such as delta-norm or $\|W_{\text{out}}\|_F$ then hill-climb to zero mean squared error.

翻译：在最近Dwarkesh Patel对John Collison和Elon Musk的播客访谈中，Jane Street提出了一个有趣的谜题：他们训练了一个神经网络，打乱了全部96层，并要求将其按原顺序重新排列。给定残差网络的未标记层及其训练数据集，我们恢复了层的精确排列顺序。该问题可分解为配对每个块的输入和输出投影（$48!$种可能性）以及对重组后的块进行排序（$48!$种可能性），总搜索空间达$(48!)^2 \approx 10^{122}$，超过可观测宇宙中的原子数量。我们证明，训练过程中的稳定性条件（如动态等距性）会使正确配对层的乘积$W_{\text{输出}} W_{\text{输入}}$呈现负对角结构，从而可利用对角占优比作为配对信号。对于排序问题，我们采用δ范数或$\|W_{\text{输出}}\|_F$等粗略代理指标进行种子初始化，随后通过爬山算法实现零均方误差。