Spatial audio, which focuses on immersive 3D sound rendering, is widely applied in the acoustic industry. One of the key problems of current spatial audio rendering methods is the lack of personalization based on different anatomies of individuals, which is essential to produce accurate sound source positions. In this work, we address this problem from an interdisciplinary perspective. The rendering of spatial audio is strongly correlated with the 3D shape of human bodies, particularly ears. To this end, we propose to achieve personalized spatial audio by reconstructing 3D human ears with single-view images. First, to benchmark the ear reconstruction task, we introduce AudioEar3D, a high-quality 3D ear dataset consisting of 112 point cloud ear scans with RGB images. To self-supervisedly train a reconstruction model, we further collect a 2D ear dataset composed of 2,000 images, each one with manual annotation of occlusion and 55 landmarks, named AudioEar2D. To our knowledge, both datasets have the largest scale and best quality of their kinds for public use. Further, we propose AudioEarM, a reconstruction method guided by a depth estimation network that is trained on synthetic data, with two loss functions tailored for ear data. Lastly, to fill the gap between the vision and acoustics community, we develop a pipeline to integrate the reconstructed ear mesh with an off-the-shelf 3D human body and simulate a personalized Head-Related Transfer Function (HRTF), which is the core of spatial audio rendering. Code and data are publicly available at https://github.com/seanywang0408/AudioEar.

翻译：空间音频专注于沉浸式3D声音渲染，在声学产业中应用广泛。当前空间音频渲染方法的关键问题之一是个体解剖结构差异导致的个性化缺失，而准确还原声源位置需要这种个性化处理。本研究从跨学科视角解决该问题。空间音频渲染与人体（尤其是耳朵）的3D形状存在强相关性。为此，我们提出通过单视角图像重建3D人耳来实现个性化空间音频。首先，为建立耳朵重建任务基准，我们构建了包含112个点云耳朵扫描及RGB图像的高质量3D耳朵数据集AudioEar3D。为实现重建模型的自监督训练，我们进一步收集了由2000张图像构成的2D耳朵数据集AudioEar2D，每张图像均包含遮挡标注和55个关键点。据我们所知，这两个数据集在同类公开数据集中规模最大、质量最优。继而，我们提出由合成数据训练的深度估计网络引导的重建方法AudioEarM，并设计了两种针对耳朵数据的损失函数。最后，为弥合视觉与声学领域的鸿沟，我们开发了将重建耳朵网格与现成3D人体模型整合的流程，并仿真出作为空间音频渲染核心的个性化头相关传递函数（HRTF）。相关代码与数据已公开于https://github.com/seanywang0408/AudioEar。