This paper considers outdoor terrain mapping using RGB images obtained from an aerial vehicle. While feature-based localization and mapping techniques deliver real-time vehicle odometry and sparse keypoint depth reconstruction, a dense model of the environment geometry and semantics (vegetation, buildings, etc.) is usually recovered offline with significant computation and storage. This paper develops a joint 2D-3D learning approach to reconstruct a local metric-semantic mesh at each camera keyframe maintained by a visual odometry algorithm. Given the estimated camera trajectory, the local meshes can be assembled into a global environment model to capture the terrain topology and semantics during online operation. A local mesh is reconstructed using an initialization and refinement stage. In the initialization stage, we estimate the mesh vertex elevation by solving a least squares problem relating the vertex barycentric coordinates to the sparse keypoint depth measurements. In the refinement stage, we associate 2D image and semantic features with the 3D mesh vertices using camera projection and apply graph convolution to refine the mesh vertex spatial coordinates and semantic features based on joint 2D and 3D supervision. Quantitative and qualitative evaluation using real aerial images show the potential of our method to support environmental monitoring and surveillance applications.

翻译：本文研究利用航空器获取的RGB图像进行户外地形建模。尽管基于特征点的定位与建图技术能够实现实时车辆里程计与稀疏关键点深度重建，但稠密的环境几何与语义（植被、建筑物等）模型通常需通过离线方式以显著的计算和存储成本恢复。本文提出一种联合二维-三维学习方法，在视觉里程计算法维护的每个相机关键帧处重建局部度量-语义网格。基于估计的相机轨迹，局部网格可在线组装成全局环境模型，以捕捉地形拓扑与语义。局部网格通过初始化和优化两个阶段重建：在初始化阶段，通过求解与稀疏关键点深度测量相关的网格顶点重心坐标最小二乘问题，估计顶点高程；在优化阶段，利用相机投影将二维图像与语义特征关联至三维网格顶点，并应用图卷积基于联合二维-三维监督信号精炼网格顶点的空间坐标与语义特征。基于真实航空图像的定性与定量评估表明，本方法在支持环境监测与 surveillance 应用方面具有潜力。