This paper presents a novel hierarchical alignment model (HAM) that learns multi-granularity visual and linguistic representations in an end-to-end manner. We extract key points and proposal points to model 3D contexts and instances, and propose point-language alignment with context modulation (PLACM) mechanism, which learns to gradually align word-level and sentence-level linguistic embeddings with visual representations, while the modulation with the visual context captures latent informative relationships. To further capture both global and local relationships, we propose a spatially multi-granular modeling scheme that applies PLACM to both global and local fields. Experimental results demonstrate the superiority of HAM, with visualized results showing that it can dynamically model fine-grained visual and linguistic representations. HAM outperforms existing methods by a significant margin and achieves state-of-the-art performance on two publicly available datasets, and won the championship in ECCV 2022 ScanRefer challenge. Code is available at~\url{https://github.com/PPjmchen/HAM}.

翻译：本文提出一种新颖的层次对齐模型（HAM），该模型以端到端方式学习多粒度视觉与语言表征。我们提取关键点和提议点以建模三维场景与实例，并提出基于上下文调制的点-语言对齐机制（PLACM），该机制通过逐步将词级与句级语言嵌入与视觉表征对齐，同时利用视觉上下文调制捕获潜在信息关系。为进一步捕获全局与局部关系，我们提出空间多粒度建模方案，将PLACM应用于全局和局部场域。实验结果表明HAM的优越性，可视化结果显示其可动态建模细粒度视觉与语言表征。HAM显著超越现有方法，在两个公开数据集上达到最先进性能，并在ECCV 2022 ScanRefer挑战赛中夺冠。代码已开源：\url{https://github.com/PPjmchen/HAM}。