Many applications require grouping instances contained in diverse document datasets into classes. Most widely used methods do not employ deep learning and do not exploit the inherently multimodal nature of documents. Notably, record linkage is typically conceptualized as a string-matching problem. This study develops CLIPPINGS, (Contrastively Linking Pooled Pre-trained Embeddings), a multimodal framework for record linkage. CLIPPINGS employs end-to-end training of symmetric vision and language bi-encoders, aligned through contrastive language-image pre-training, to learn a metric space where the pooled image-text representation for a given instance is close to representations in the same class and distant from representations in different classes. At inference time, instances can be linked by retrieving their nearest neighbor from an offline exemplar embedding index or by clustering their representations. The study examines two challenging applications: constructing comprehensive supply chains for mid-20th century Japan through linking firm level financial records - with each firm name represented by its crop in the document image and the corresponding OCR - and detecting which image-caption pairs in a massive corpus of historical U.S. newspapers came from the same underlying photo wire source. CLIPPINGS outperforms widely used string matching methods by a wide margin and also outperforms unimodal methods. Moreover, a purely self-supervised model trained on only image-OCR pairs also outperforms popular string-matching methods without requiring any labels.

翻译：许多应用需要将分散于不同文档数据集中的实例划分为类别。当前最常用的方法并未采用深度学习，也未利用文档固有的多模态特性。值得注意的是，记录链接通常被概念化为字符串匹配问题。本研究提出了CLIPPINGS（对比式链接池化预训练嵌入）框架，这是一种用于记录链接的多模态方法。CLIPPINGS通过端到端训练对称的视觉与语言双编码器，并借助对比式语言-图像预训练进行对齐，从而学习一个度量空间，在该空间中，同一类别的实例对应的池化图像-文本表示相互靠近，而不同类别的实例表示相互远离。在推理阶段，可通过从离线示例嵌入索引中检索最近邻或对表示进行聚类来实现实例链接。本研究考察了两个具有挑战性的应用场景：一是通过链接企业级财务记录构建20世纪中期日本的综合供应链（其中每个企业名称由文档图像中的裁剪区域及其对应OCR表示），二是检测海量历史美国报纸语料库中哪些图像-标题对源自同一摄影通讯社源。CLIPPINGS在性能上大幅超越广泛使用的字符串匹配方法，同时也优于单模态方法。此外，仅使用图像-OCR对训练的纯自监督模型在无需任何标注的情况下，同样优于流行的字符串匹配方法。