Autonomous vehicles are equipped with a multi-modal sensor setup to enable the car to drive safely. The initial calibration of such perception sensors is a highly matured topic and is routinely done in an automated factory environment. However, an intriguing question arises on how to maintain the calibration quality throughout the vehicle's operating duration. Another challenge is to calibrate multiple sensors jointly to ensure no propagation of systemic errors. In this paper, we propose CaLiCa, an end-to-end deep self-calibration network which addresses the automatic calibration problem for pinhole camera and Lidar. We jointly predict the camera intrinsic parameters (focal length and distortion) as well as Lidar-Camera extrinsic parameters (rotation and translation), by regressing feature correlation between the camera image and the Lidar point cloud. The network is arranged in a Siamese-twin structure to constrain the network features learning to a mutually shared feature in both point cloud and camera (Lidar-camera constraint). Evaluation using KITTI datasets shows that we achieve 0.154 {\deg} and 0.059 m accuracy with a reprojection error of 0.028 pixel with a single-pass inference. We also provide an ablative study of how our end-to-end learning architecture offers lower terminal loss (21% decrease in rotation loss) compared to isolated calibration

翻译：自动驾驶车辆配备多模态传感器组以确保安全行驶。这类感知传感器的初始标定技术已高度成熟，通常在自动化工厂环境中例行完成。然而，如何在整个车辆运行周期内保持标定质量仍是一个引人深思的问题。另一挑战在于如何对多个传感器进行联合标定，以避免系统误差的传播。本文提出CaLiCa——一种面向针孔相机与激光雷达的端到端深度自标定网络。通过回归相机图像与激光雷达点云之间的特征相关性，我们联合预测相机内参（焦距与畸变）及激光雷达-相机外参（旋转与平移）。该网络采用孪生结构设计，约束网络特征学习至点云与相机的互共享特征空间（激光雷达-相机约束）。基于KITTI数据集的评估表明，单次推理可实现0.154°与0.059米的精度，重投影误差为0.028像素。我们还通过消融研究证明，与孤立标定相比，端到端学习架构能实现更低终端损失（旋转损失降低21%）。