ImageNet was famously created from Flickr image search results. What if we recreated ImageNet instead by searching the massive LAION dataset based on image captions alone? In this work, we carry out this counterfactual investigation. We find that the resulting ImageNet recreation, which we call LAIONet, looks distinctly unlike the original. Specifically, the intra-class similarity of images in the original ImageNet is dramatically higher than it is for LAIONet. Consequently, models trained on ImageNet perform significantly worse on LAIONet. We propose a rigorous explanation for the discrepancy in terms of a subtle, yet important, difference in two plausible causal data-generating processes for the respective datasets, that we support with systematic experimentation. In a nutshell, searching based on an image caption alone creates an information bottleneck that mitigates the selection bias otherwise present in image-based filtering. Our explanation formalizes a long-held intuition in the community that ImageNet images are stereotypical, unnatural, and overly simple representations of the class category. At the same time, it provides a simple and actionable takeaway for future dataset creation efforts.

翻译：ImageNet的创建众所周知源于Flickr图像搜索结果。如果我们仅依据图像描述，通过搜索海量LAION数据集来重建ImageNet，结果会如何？本研究正是对这一反事实假设的实证探索。我们发现由此生成的重建数据集（称为LAIONet）与原始数据集存在显著差异。具体而言，原始ImageNet中图像的类内相似度明显高于LAIONet。因此，在ImageNet上训练的模型在LAIONet上的表现显著下降。我们通过系统化实验验证，提出一种严谨的解释：这种差异源于两个数据集各自可能的数据生成过程中存在微妙却关键的区别。简而言之，仅基于图像描述的搜索会形成信息瓶颈，从而缓解基于图像过滤时可能存在的选择偏差。我们的解释形式化地验证了学界长期持有的直觉：ImageNet图像具有刻板化、非自然且过度简化的类别表征特征。同时，这一发现为未来数据集构建工作提供了简明且可操作的启示。