Visual and linguistic concepts naturally organize themselves in a hierarchy, where a textual concept "dog" entails all images that contain dogs. Despite being intuitive, current large-scale vision and language models such as CLIP do not explicitly capture such hierarchy. We propose MERU, a contrastive model that yields hyperbolic representations of images and text. Hyperbolic spaces have suitable geometric properties to embed tree-like data, so MERU can better capture the underlying hierarchy in image-text datasets. Our results show that MERU learns a highly interpretable and structured representation space while being competitive with CLIP's performance on standard multi-modal tasks like image classification and image-text retrieval. Our code and models are available at https://www.github.com/facebookresearch/meru

翻译：视觉与语言概念天然地以层级结构组织，其中文本概念"狗"涵盖所有包含狗的图像。尽管这一特性直观合理，但当前大规模视觉-语言模型（如CLIP）并未显式捕捉此类层级关系。我们提出MERU——一种生成图像与文本双曲表示的对比学习模型。双曲空间具有嵌入树状数据的几何特性，因此MERU能够更好地捕捉图像-文本数据中的潜在层级结构。实验结果表明，MERU在图像分类与图像-文本检索等标准多模态任务中与CLIP性能相当，同时学习到高度可解释且结构化的表示空间。我们的代码与模型已开源至https://www.github.com/facebookresearch/meru