We present an approach for multimodal pathology image search, using dynamic time warping (DTW) on Variational Autoencoder (VAE) latent space that is fed into a ranked choice voting scheme to retrieve multiplexed immunofluorescent imaging (mIF) that is most similar to a query H&E slide. Through training the VAE and applying DTW, we align and compare mIF and H&E slides. Our method improves differential diagnosis and therapeutic decisions by integrating morphological H&E data with immunophenotyping from mIF, providing clinicians a rich perspective of disease states. This facilitates an understanding of the spatial relationships in tissue samples and could revolutionize the diagnostic process, enhancing precision and enabling personalized therapy selection. Our technique demonstrates feasibility using colorectal cancer and healthy tonsil samples. An exhaustive ablation study was conducted on a search engine designed to explore the correlation between multiplexed Immunofluorescence (mIF) and Hematoxylin and Eosin (H&E) staining, in order to validate its ability to map these distinct modalities into a unified vector space. Despite extreme class imbalance, the system demonstrated robustness and utility by returning similar results across various data features, which suggests potential for future use in multimodal histopathology data analysis.

翻译：我们提出了一种多模态病理图像检索方法，该方法在变分自编码器（VAE）潜空间上应用动态时间规整（DTW），并将其输入到排名选择投票机制中，以检索与查询H&E切片最相似的多重免疫荧光成像（mIF）。通过训练VAE并应用DTW，我们对mIF与H&E切片进行对齐和比较。该方法通过整合H&E的形态学数据与mIF的免疫表型信息，改善了鉴别诊断与治疗决策，为临床医生提供了关于疾病状态的丰富视角。这有助于理解组织样本中的空间关联性，并可能革新诊断流程，提升精确度，从而实现个性化治疗方案的选择。我们利用结直肠癌和健康扁桃体样本验证了该技术的可行性。针对一个旨在探索多重免疫荧光（mIF）与苏木精-伊红（H&E）染色之间相关性的检索系统，我们开展了详尽的消融实验，以验证其将这些不同模态映射到统一向量空间的能力。尽管存在极端的类别不平衡问题，该系统仍能在多种数据特征上返回相似结果，展现出稳健性与实用性，这为其未来在多模态组织病理学数据分析中的应用提供了潜力。