Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multimodal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.

翻译：生物多样性测量对于理解生态系统健康至关重要。尽管先前的研究已分别开发了用于照片图像和DNA分类的机器学习模型，但本工作引入了一种结合两者的多模态方法，利用CLIP风格的对比学习，将图像、条形码DNA以及基于文本的分类标签表示对齐到一个统一的嵌入空间中。这使得无需针对特定任务进行微调即可准确分类已知和未知的昆虫物种，并首次利用对比学习融合DNA与图像数据。在零样本学习任务中，我们的方法在准确率上超越了以往的单模态方法超过8%，展示了其在生物多样性研究中的有效性。