Visually rich documents (VRD) are physical/digital documents that utilize visual cues to augment their semantics. The information contained in these documents are often incomplete. Existing works that enable automated querying on VRDs do not take this aspect into account. Consequently, they support a limited set of queries. In this paper, we describe Juno -- a multimodal framework that identifies a set of tuples from a relational database to augment an incomplete VRD with supplementary information. Our main contribution in this is an end-to-end-trainable neural network with bi-directional attention that executes this cross-modal entity matching task without any prior knowledge about the document type or the underlying database-schema. Exhaustive experiments on two heteroegeneous datasets show that Juno outperforms state-of-the-art baselines by more than 6% in F1-score, while reducing the amount of human-effort in its workflow by more than 80%. To the best of our knowledge, ours is the first work that investigates the incompleteness of VRDs and proposes a robust framework to address it in a seamless way.

翻译：丰富视觉文档（VRD）是指利用视觉线索增强语义的物理或数字文档。这些文档所包含的信息往往是不完整的。现有支持对VRD进行自动查询的研究未考虑这一方面，因此仅支持有限的查询类型。本文提出Juno——一种多模态框架，能从关系数据库中识别出一组元组，用补充信息来完善不完整的VRD。我们在此项研究中的主要贡献是：提出一种具有双向注意力机制的端到端可训练神经网络，能在无需预先了解文档类型或底层数据库模式的情况下执行跨模态实体匹配任务。在两个异构数据集上的详尽实验表明，Juno在F1分数上比现有最优基线方法高出6%以上，同时将工作流中所需的人工工作量减少了80%以上。据我们所知，这是首个探究VRD不完整性并为此提出稳健无缝解决方案的研究工作。