The Platonic Representation Hypothesis suggests that neural networks trained on different modalities (e.g., text and images) align and eventually converge toward the same representation of reality. If true, this has significant implications for whether modality choice matters at all. We show that the experimental evidence for this hypothesis is fragile and depends critically on the evaluation regime. Alignment is measured using mutual nearest neighbors on small datasets ($\approx$1K samples) and degrades substantially as the dataset is scaled to millions of samples. The alignment that remains between model representations reflects coarse semantic overlap rather than consistent fine-grained structure. Moreover, the evaluations in Huh et al. are done in a one-to-one image-caption setting, a constraint that breaks down in realistic many-to-many settings and further reduces alignment. We also find that the reported trend of stronger language models increasingly aligning with vision does not appear to hold for newer models. Overall, our findings suggest that the current evidence for cross-modal representational convergence is considerably weaker than subsequent works have taken it to be. Models trained on different modalities may learn equally rich representations of the world, just not the same one.

翻译：柏拉图表征假说认为，不同模态（如文本与图像）训练的神经网络会逐渐对齐并最终收敛至相同的现实表征。若该假说成立，将对模态选择是否具有根本性影响产生重要启示。我们通过实验表明，该假说的证据具有脆弱性，且严重依赖于评估机制。现有研究使用小规模数据集（约1000个样本）进行互近邻对齐度量，当数据集规模扩展至百万级样本时，对齐程度显著下降。模型表征间残存的对齐反映的是粗粒度语义重叠，而非一致的细粒度结构。此外，Huh等人的评估均基于一对一图像-文本配对设定，这种约束在面对现实多对多场景时崩溃，进一步降低了对齐程度。我们还发现，更强的语言模型与视觉模型同步增强的趋势在新模型中似乎不再成立。总体而言，本研究表明跨模态表征收敛性的现有证据实际上远弱于后续研究认为的程度。不同模态训练的模型或许能习得同样丰富的世界表征，但并非相同的表征。