Finding scientifically interesting phenomena through slow manual labeling campaigns severely limits our ability to explore the billions of galaxy images produced by telescopes. In this work, we develop a pipeline to create a semantic search engine from completely unlabeled image data. Our method leverages Vision-Language Models (VLMs) to generate descriptions for galaxy images, then contrastively aligns a pre-trained astronomy foundation model with these embedded descriptions to produce searchable embeddings at scale. We find that current VLMs provide descriptions that are sufficiently informative to train a semantic search model that outperforms direct image similarity search. Our model, AION-Search, achieves state-of-the-art zero-shot performance on finding rare phenomena despite training on randomly selected images with no deliberate curation for rare cases. Furthermore, we introduce a VLM-based re-ranking method that nearly doubles the recall for our most challenging targets in the top-100 results. For the first time, AION-Search enables flexible semantic search for over 100 million galaxy images, enabling discovery from previously infeasible searches, including the identification of 36 new extragalactic stellar stream candidates. More broadly, our work provides an approach for making large, unlabeled scientific image archives semantically searchable, expanding data exploration capabilities in fields from Earth observation to microscopy. The code, data, and app are publicly available at https://github.com/NolanKoblischke/AION-Search

翻译：通过缓慢的手动标注活动来寻找科学上有趣的现象，严重限制了我们对望远镜产生的数十亿星系图像的探索能力。在这项工作中，我们开发了一个流程，从完全未标记的图像数据中构建一个语义搜索引擎。我们的方法利用视觉-语言模型（VLM）为星系图像生成描述，然后通过对比学习将预训练的天文学基础模型与这些嵌入描述对齐，以大规模生成可搜索的嵌入向量。我们发现，当前的VLM提供的描述信息量足以训练一个语义搜索模型，其性能优于直接的图像相似性搜索。我们的模型AION-Search在寻找罕见现象方面实现了最先进的零样本性能，尽管其训练数据是随机选择的图像，并未针对罕见案例进行刻意筛选。此外，我们引入了一种基于VLM的重排序方法，该方法在排名前100的结果中，使我们最具挑战性目标的召回率几乎翻倍。AION-Search首次实现了对超过1亿张星系图像的灵活语义搜索，能够从之前不可行的搜索中实现发现，包括识别出36个新的河外星系星流候选体。更广泛地说，我们的工作为将大规模、未标记的科学图像档案变得语义可搜索提供了一种方法，扩展了从地球观测到显微镜学等领域的数据探索能力。代码、数据和应用程序已在https://github.com/NolanKoblischke/AION-Search公开提供。