Vision-and-Language Navigation (VLN) is a challenging task in which an agent needs to follow a language-specified path to reach a target destination. The goal gets even harder as the actions available to the agent get simpler and move towards low-level, atomic interactions with the environment. This setting takes the name of low-level VLN. In this paper, we strive for the creation of an agent able to tackle three key issues: multi-modality, long-term dependencies, and adaptability towards different locomotive settings. To that end, we devise "Perceive, Transform, and Act" (PTA): a fully-attentive VLN architecture that leaves the recurrent approach behind and the first Transformer-like architecture incorporating three different modalities - natural language, images, and low-level actions for the agent control. In particular, we adopt an early fusion strategy to merge lingual and visual information efficiently in our encoder. We then propose to refine the decoding phase with a late fusion extension between the agent's history of actions and the perceptual modalities. We experimentally validate our model on two datasets: PTA achieves promising results in low-level VLN on R2R and achieves good performance in the recently proposed R4R benchmark. Our code is publicly available at https://github.com/aimagelab/perceive-transform-and-act.

翻译：视觉和语言导航(VLN)是一项具有挑战性的任务,在这个任务中,一个代理机构需要遵循一种语言指定的路径才能达到目标目的地。当该代理机构可以采取的行动变得更加简单,并朝着低水平的原子与环境互动的方向发展时,目标变得更加困难。这个设置的名称是低水平VLN。在本文中,我们努力创建一个能够解决三个关键问题的代理机构:多模式、长期依赖性和适应不同车头环境的适应性。为此,我们设计了“循环、变换和动作”(PTA):一个完全敏捷的VLN结构,将经常性的方法留在后面,而第一个变换式结构则包含三种不同的方式——自然语言、图像和低层次的代理控制行动。特别是,我们采取了早期融合战略,将语言和视觉信息有效地结合到我们的编码中。我们然后建议改进解码阶段,在代理人的行动史和感官模式2(PTA)之间,我们实验性地验证了我们两个有希望的模型:Rabrbral-Rimal 的模型,最近在Rab-Rimalal 上实现了我们提出的低级的成绩基准。