Contemporary autonomous vehicle (AV) benchmarks have advanced techniques for training 3D detectors, particularly on large-scale lidar data. Surprisingly, although semantic class labels naturally follow a long-tailed distribution, contemporary benchmarks focus on only a few common classes (e.g., pedestrian and car) and neglect many rare classes in-the-tail (e.g., debris and stroller). However, AVs must still detect rare classes to ensure safe operation. Moreover, semantic classes are often organized within a hierarchy, e.g., tail classes such as child and construction-worker are arguably subclasses of pedestrian. However, such hierarchical relationships are often ignored, which may lead to misleading estimates of performance and missed opportunities for algorithmic innovation. We address these challenges by formally studying the problem of Long-Tailed 3D Detection (LT3D), which evaluates on all classes, including those in-the-tail. We evaluate and innovate upon popular 3D detection codebases, such as CenterPoint and PointPillars, adapting them for LT3D. We develop hierarchical losses that promote feature sharing across common-vs-rare classes, as well as improved detection metrics that award partial credit to "reasonable" mistakes respecting the hierarchy (e.g., mistaking a child for an adult). Finally, we point out that fine-grained tail class accuracy is particularly improved via multimodal fusion of RGB images with LiDAR; simply put, small fine-grained classes are challenging to identify from sparse (lidar) geometry alone, suggesting that multimodal cues are crucial to long-tailed 3D detection. Our modifications improve accuracy by 5% AP on average for all classes, and dramatically improve AP for rare classes (e.g., stroller AP improves from 3.6 to 31.6)! Our code is available at https://github.com/neeharperi/LT3D

翻译：当代自动驾驶（AV）基准测试推动了三维检测器训练技术的进步，尤其是在大规模激光雷达数据上。令人惊讶的是，尽管语义类别标签自然遵循长尾分布，当代基准测试仅关注少数常见类别（如行人和汽车），而忽略了尾部中的许多稀有类别（如碎片和婴儿车）。然而，自动驾驶汽车仍需检测稀有类别以确保安全运行。此外，语义类别通常组织在层次结构中，例如，儿童和建筑工人等尾部类别可以说是行人的子类。然而，这种层次关系往往被忽略，可能导致对性能的误导性估计，并错失算法创新的机会。我们通过正式研究长尾三维检测（LT3D）问题来应对这些挑战，该问题评估所有类别，包括尾部类别。我们在流行的三维检测代码库（如CenterPoint和PointPillars）上进行评估和创新，使其适应LT3D。我们开发了层次化损失函数，以促进常见与稀有类别间的特征共享，以及改进的检测指标，对符合层次结构的“合理”错误（例如，将儿童误认为成人）给予部分分数。最后，我们指出，通过RGB图像与激光雷达的多模态融合，细粒度尾部类别的准确性尤其得到提升；简单来说，小尺寸细粒度类别难以仅从稀疏（激光雷达）几何信息中识别，这表明多模态线索对长尾三维检测至关重要。我们的改进使所有类别的平均AP提高了5%，并显著提升了稀有类别的AP（例如，婴儿车的AP从3.6提高到31.6）！我们的代码可在https://github.com/neeharperi/LT3D获取。